Skip to content

Incident Response

When something is on fire

  1. Check @DownRabbitbot Telegram channel — has it already alerted? Read the message.
  2. Check Uptime Kuma at the SSH-tunnel URL — which monitors are red?
  3. SSH to the VPS: ssh vps
  4. docker ps — what is down or restarting?
  5. docker logs <container> --tail 200 — what is the immediate error?

Common cases

A container is in Restarting state

docker logs <name> --tail 100
docker inspect <name> | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['State'])"

Most common cause: failed migration, missing env var, port collision. Do not force-remove the container — find the root cause first.

Disk full

df -h
docker system df
docker system prune       # safe: removes only stopped containers and dangling images
docker system prune -a    # DESTRUCTIVE: removes all unused images (ask first)

Build cache is usually the biggest waste — clearing it costs nothing but the next build's speed.

Postgres slow / locked

docker exec infra-postgres psql -U postgres -c "SELECT pid, query, state, wait_event FROM pg_stat_activity WHERE state != 'idle';"

If a query is wedged, capture its pid and pg_terminate_backend(pid). Do not restart Postgres unless absolutely necessary — it interrupts every app.

nginx returns 502

Means an upstream app is down or not listening. Check docker ps for the upstream. Check nginx logs: 

docker logs infra-nginx --tail 50

TLS cert about to expire

infra-certbot should auto-renew. If it has not, check its logs and run the renewal manually inside the container.

LiteLLM cost cap hit

Look at the litellm dashboard. The hard cap (currently $25/mo) blocks new requests until the next billing cycle or until the cap is raised. Do not raise the cap without an ADR.

Escalation path

Level Who When
1 On-call (you) First 15 minutes
2 Project owner from index.md After 15 minutes if unresolved
3 João Anything touching production data, money, or auth

After the incident

Within 24 hours of any incident that caused user-visible impact:

  1. Write the post-mortem in runbooks/post-mortems/ — short is fine
  2. Add to errors/REGISTRY.md if a new failure mode was discovered
  3. Open a Vikunja task for any prevention work identified
  4. If a decision needs to be made (rollback policy change, etc.), write an ADR