Runbook: Apache AGE Graph Layer — Maintenance & Migration

Last updated: 2026-06-16
Relates to: ADR-0007 — Apache AGE as Property Graph Layer

---

Overview

Apache AGE 1.5.0 runs inside infra-postgres (image localhost/infra-postgres-age:16-age1.5.0).
Graph name: down_rabbit_hole_graph
Co-resident with pgvector 0.8.2 — both loaded via shared_preload_libraries = 'age'.

---

1. Verify AGE is healthy

docker exec infra-postgres psql -U litellm -d down_rabbit_hole -c "
  LOAD 'age';
  SET search_path = ag_catalog, '\$user', public;
  SELECT * FROM cypher('down_rabbit_hole_graph', \$\$
    MATCH (n) RETURN count(n) AS nodes
  \$\$) AS (nodes agtype);"

Expected: returns row count of graph vertices. If you see ERROR: extension "age" is not available, see §4.

---

2. Rebuild the graph from relational data

Run via the gateway skill (preferred):

curl -s -X POST http://localhost:8095/skills/brain.graph-rebuild/run \
  -H "Content-Type: application/json" \
  -d '{"scope": "all"}' | python3 -m json.tool

Or directly via PostgreSQL if the skill is unavailable:

docker exec infra-postgres psql -U litellm -d down_rabbit_hole -c "
  SELECT brain_rebuild_graph();"

Monitor progress:

docker exec infra-postgres psql -U litellm -d down_rabbit_hole -c "
  LOAD 'age';
  SET search_path = ag_catalog, '\$user', public;
  SELECT label_name, count(*) FROM cypher('down_rabbit_hole_graph', \$\$
    MATCH (n) RETURN labels(n)[0] AS label_name, count(n)
  \$\$) AS (label_name agtype, count agtype)
  GROUP BY 1;"

---

3. Upgrade AGE version

Do not upgrade in-place. Build a new image, migrate data, swap.

3.1 Build new image

# On VPS, in /opt/infrastructure/postgres-age/
docker build --build-arg AGE_VERSION=x.y.z \
  -t localhost/infra-postgres-age:16-agex.y.z .

3.2 Backup current graph

docker exec infra-postgres pg_dump -U litellm \
  --schema=ag_catalog down_rabbit_hole \
  > /opt/backups/ag_catalog_$(date +%Y%m%d).sql

3.3 Export graph to JSON (human-readable backup)

curl -s -X POST http://localhost:8095/skills/brain.graph-rebuild/run \
  -H "Content-Type: application/json" \
  -d '{"scope": "export", "format": "json"}' > /opt/backups/graph_export_$(date +%Y%m%d).json

3.4 Swap image

Edit /opt/infrastructure/docker-compose.yml:

infra-postgres:
  image: localhost/infra-postgres-age:16-agex.y.z   # update here

Then:

docker compose -f /opt/infrastructure/docker-compose.yml up -d --no-deps infra-postgres
sleep 10
# Verify AGE loaded:
docker exec infra-postgres psql -U litellm -d down_rabbit_hole -c "LOAD 'age';"

3.5 Rebuild graph on new version

Run §2 above. The ag_catalog schema is recreated by AGE on first CREATE EXTENSION age.

---

4. Recovery — AGE not loading

Symptom: ERROR: extension "age" is not available
Cause: Container running wrong image (without AGE compiled in) or shared_preload_libraries not set.

# Verify image
docker inspect infra-postgres --format "{{.Config.Image}}"
# Expected: localhost/infra-postgres-age:16-age1.5.0

# Verify shared_preload_libraries
docker exec infra-postgres psql -U litellm -c "SHOW shared_preload_libraries;"
# Expected: includes 'age'

If wrong image, recreate:

docker compose -f /opt/infrastructure/docker-compose.yml up -d --no-deps infra-postgres

---

5. Recovery — graph data missing after restart

AGE graph data persists in the ag_catalog schema inside the down_rabbit_hole database, which is on the infra-postgres-data named volume. Data survives container restarts. If the volume was inadvertently removed, rebuild from relational data (§2) — the graph is fully derivable from nodes + node_links tables.

---

6. Key facts

Item	Value
PostgreSQL version	16.14
AGE version	1.5.0
pgvector version	0.8.2
Graph name	down_rabbit_hole_graph
Docker image	localhost/infra-postgres-age:16-age1.5.0
Rebuild skill	brain.graph-rebuild
Graph vertex labels	BrainNode, TaxonomyTerm, Entity
Graph edge labels	RELATED_TO, CONTRADICTS, PART_OF (and others)