Skip to content

Runbook — Restore gamine_cms Database from Backup

Scope: gamine-cms project — Wagtail CMS for gamine.org
Frequency: On incident (accidental drop, corruption, failed migration)
Estimated time: 15–30 min depending on backup size

Prerequisites

  • SSH access to VPS (ssh vps)
  • Backup file at /opt/backups/postgres/gamine_cms_<DATE>.sql.gz
  • gamine-cms container running (or stopped, depending on step)
  • litellm PostgreSQL user credentials (from Vaultwarden → Infrastructure → PostgreSQL)

Step 1 — Identify the correct backup

ls -lh /opt/backups/postgres/gamine_cms_*.sql.gz | tail -10
# Pick the most recent clean backup — check size (healthy backup is > 50KB)
du -h /opt/backups/postgres/gamine_cms_2026-05-21.sql.gz

⚠️ If all backups are tiny (< 1KB), the DB was never populated or backup.sh was broken. Check /opt/backups/postgres/ for the newest non-empty file.

Step 2 — Stop gamine-cms to prevent writes during restore

cd /opt/rabbithall-compose   # or wherever gamine-cms compose lives when deployed
docker compose stop gamine-cms

Step 3 — Drop and recreate the database

docker exec infra-postgres psql -U litellm -d postgres -c "
  SELECT pg_terminate_backend(pid)
  FROM pg_stat_activity
  WHERE datname = 'gamine_cms' AND pid <> pg_backend_pid();
"
docker exec infra-postgres psql -U litellm -d postgres -c "DROP DATABASE IF EXISTS gamine_cms;"
docker exec infra-postgres psql -U litellm -d postgres -c "CREATE DATABASE gamine_cms OWNER litellm;"

Step 4 — Restore from backup

BACKUP=/opt/backups/postgres/gamine_cms_2026-05-21.sql.gz  # adjust date

gunzip -c "$BACKUP" | docker exec -i infra-postgres psql -U litellm -d gamine_cms

Verify row counts after restore:

docker exec infra-postgres psql -U litellm -d gamine_cms -c "
  SELECT schemaname, tablename, n_live_tup
  FROM pg_stat_user_tables
  ORDER BY n_live_tup DESC
  LIMIT 20;
"

Step 5 — Run pending Django migrations (if restoring to older schema)

cd /opt/gamine-cms   # when deployed
docker compose run --rm gamine-cms python manage.py migrate --run-syncdb

Step 6 — Restart and verify

docker compose start gamine-cms
sleep 5
curl -s https://gamine.org/health/ | python3 -m json.tool
# Expected: {"status": "ok", "version": "x.y.z", "db": "ok"}

Check Wagtail admin is accessible:

curl -o /dev/null -sw "%{http_code}" https://gamine.org/cms-admin/
# Expected: 200 or 302 (redirect to login)

Step 7 — Notify

Send Telegram alert confirming restore complete:

curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
  -d chat_id="${TELEGRAM_CHAT_ID}" \
  -d text="✅ gamine_cms DB restored from backup $(date -u +%Y-%m-%dT%H:%M)Z. Verify: https://gamine.org"

Rollback procedure (if restore made things worse)

If the restore fails partway through and the DB is in an inconsistent state:

  1. Drop gamine_cms again (Step 3)
  2. Try the next-oldest backup
  3. If all backups are bad → escalate to João; run python manage.py migrate --fake-initial from a fresh DB and re-import content from gamine-brain snapshots
  • backup.sh at /opt/backup.sh — runs daily, keeps 30 days of postgres dumps
  • Vikunja #665 — this runbook
  • ADR-006 (adr-006-wagtail-for-gamine-cms.md) — why Wagtail was chosen
  • Wagtail docs: https://docs.wagtail.org/en/stable/