Skip to content

Knowledge Harvest Loop — Overnight Implementation Manual

You are a fresh Claude Code session with no memory of the design conversation. This manual is self-contained. Read the Prime Directive and Environment sections fully before doing anything. The architecture you are implementing is specified in the companion file: specs/brain/2026-06-20-knowledge-harvest-loop.md — read it first.

0. PRIME DIRECTIVE — read before acting

You are implementing the Knowledge Harvest Loop unattended, overnight. The owner (João) is asleep and cannot answer questions. Therefore:

You MAY do autonomously (safe, additive, reversible, no real cost)

  • Read code, DB schema, configs.
  • Create new files: gateway skills, migration SQL, admin templates, tests.
  • Apply additive DB changes only (CREATE TABLE IF NOT EXISTS, ADD COLUMN IF NOT EXISTS) with a written reversible down script.
  • Restart jb-skills-gateway to load new skills (verify healthy afterwards).
  • Run tests in dry_run mode only (no real web research, no paid LLM batches; trivial single LLM calls for a unit test are acceptable).
  • Commit work to a new feature branch in each affected repo.

You MUST NOT do (HARD STOP — leave for João's morning approval)

  • Enable any cron / scheduler.
  • Set or raise any budget value.
  • Run the first real (paid) harvest (real Deeper Research run that spends money).
  • Send any Telegram / email / public message (test notify.push in dry-run only).
  • Deploy the admin UI tab to the live rabbithall hub (write the files only).
  • Merge to main / push to main / promote to production. Push feature branches only.
  • Any destructive op: DROP, DELETE, TRUNCATE, overwriting existing skills/files, force-recreate of shared containers.

Operating rules

  • Plan fully, apply once. Never trial-and-error on production containers. Reason the whole change, then make it in one pass, then verify.
  • Verify after every phase. If a verification fails, STOP that phase, record it in the morning report, and only continue with later phases that are independent and safe. Do not cascade on a failure.
  • No silent caps. If you skip or truncate anything, write it in the report.
  • Keep all generated UI light theme only (the owner is colorblind — dark backgrounds are unusable). Copy the CSS variables from /opt/rabbithall/templates/admin_dashboard.html.
  • Work additive: never edit tcn.ingest-article or other live skills in place — add new code paths behind flags.

1. Environment (facts you need; you have no memory)

  • VPS access: ssh vps (root). Primary host 76.13.44.83.
  • Gateway: container jb-skills-gateway, uvicorn --workers 3. Skills live in /opt/jb-skills-gateway/skills/<namespace>/<skill>/main.py (+ empty __init__.py), bind-mounted (host edits are live). Internal base URL http://localhost:8095. To load new/changed skills: docker restart jb-skills-gateway (NOT /reload). A skill needs SKILL_META = {...} and def run(payload: dict) or it fails silently with no log.
  • DB: docker exec infra-postgres psql -U litellm -d down_rabbit_hole. Relevant tables: taxonomy_terms, search_topology (last_seen = last search, ordered ASC for stale-first), discovery_queue (content candidates, domain-agnostic), nodes (embedding_768, summary, main_takeaways[], metadata jsonb), brain_staging. Apache AGE graph: down_rabbit_hole_graph (Entities, MENTIONS, LINKS_TO).
  • AI: ALL via the LiteLLM proxy — never call providers directly. Chat tiers include brain-summarise, brain-classify. Embeddings: gemini/gemini-embedding-001, 768-dim, via LiteLLM /embeddings.
  • Existing reusable skills: tcn.six-eyes-scorer (POST /skills/tcn.six-eyes-scorer/run; thresholds <3 discard / 3–6 quarantine / ≥6 auto-ingest), tcn.deep-research (POSTs to dr_orchestrator:8200), brain.content-curator (ingest + cosine MAX(1 - (embedding_768 <=> v))), brain.ontology-enricher (extract entities → propose new terms), notify.push (category/severity; only action+critical → Telegram).
  • Admin UI: rabbithall, hub at /admin/curation. CRITICAL deploy path: the real compose is /opt/rabbithall-compose/ (NOT /opt/rabbithall/docker-compose.yml — that one is obsolete and using it takes the hub down). Templates+static are bind-mounted; app.py is baked (deploy via rh-deploy). For THIS overnight run you only write the template/route into the repo — you do not deploy.
  • Git/Forgejo: SSH remote ssh://git@git.joaoluisbrazao.cloud:2222/joaobrazao/<repo>.git. Repos: jb-skills-gateway, rabbithall (confirm working-tree location on VPS, likely /opt/jb-skills-gateway and /opt/rabbithall). Branch naming: feature/harvest-loop.
  • Tasks: Vikunja DEV project_id=4. Insert via docker exec infra-postgres psql -U litellm -d vikunja -c "INSERT INTO tasks (title,description,project_id,index,repeat_mode,created,updated,created_by_id) VALUES (...)", created_by_id=1, index = current max+1.
  • Shell gotcha: the remote command runner rejects backslashes/newlines in some cases — prefer single-line commands without \ escapes (e.g. unescaped dots in sed/grep patterns).

2. Phases (execute in order; verify each)

Phase 0 — Preflight (read-only) + branch

  1. Read the spec specs/brain/2026-06-20-knowledge-harvest-loop.md.
  2. ssh vps and confirm: jb-skills-gateway healthy; infra-postgres reachable; tables taxonomy_terms, search_topology, discovery_queue, nodes exist; LiteLLM reachable (/embeddings returns 200 for a trivial input).
  3. In the jb-skills-gateway working tree on the VPS: git checkout -b feature/harvest-loop (do NOT touch main). Verify: branch created; all preflight checks green. If any fails → STOP, report.

Phase 1 — DB schema (additive, reversible)

Write migrations/harvest_001_up.sql and harvest_001_down.sql: - CREATE TABLE IF NOT EXISTS harvest_targets ( id uuid PK default gen_random_uuid(), term text, topology text, domain varchar(100), destination jsonb, status varchar(20) default 'pending', depth int default 0, parent_target_id uuid, priority int default 100, enqueued_at timestamptz default now(), started_at timestamptz, finished_at timestamptz, tokens_spent int default 0, result_summary text, created_at timestamptz default now(), updated_at timestamptz default now(), deleted_at timestamptz ); partial unique index on (term, domain) where status in ('pending','researching'). - ALTER TABLE search_topology ADD COLUMN IF NOT EXISTS last_novelty_at timestamptz, ADD COLUMN IF NOT EXISTS check_interval int. - ALTER TABLE nodes ADD COLUMN IF NOT EXISTS new_events jsonb, ADD COLUMN IF NOT EXISTS potential_consequences jsonb, ADD COLUMN IF NOT EXISTS connections jsonb (or store these in existing metadata jsonb — pick one and be consistent; columns preferred for queryability). Apply the up to down_rabbit_hole (additive only). Verify: \d harvest_targets shows the table; columns added. down script written and reviewed (do not run it).

Phase 2 — harvest.enqueue-term (public API)

Create /opt/jb-skills-gateway/skills/harvest/enqueue-term/{__init__.py,main.py}. SKILL_META name harvest.enqueue-term. Accept {term, topology, domain, destination, depth?, parent_target_id?}. Dedup against harvest_targets (pending/researching) and skip if the term already exists in taxonomy_terms for P1. Insert a pending row. docker restart jb-skills-gateway. Verify: call the skill with a test payload (dry_run:true inserts nothing, just validates); confirm it is registered and a non-dry call inserts exactly one row and dedups on a second identical call. Clean up the test row.

Phase 3 — harvest.worker (paced executor, NIGHT = dry)

Create harvest/worker/. Logic: check budget (Phase 4) → pull up to K pending targets ordered by priority, depth ASC, enqueued_at → for each, call Deeper Research → normalize → distill (Phase 6) → novelty gate (Phase 5) → Six Eyes → route (<3 discard / 3–6 quarantine / ≥6 ingest) → update search_topology.last_seen/last_novelty_at + recompute cadence (Phase 7). Implement a dry_run mode that does NOT call paid Deeper Research — it must exercise the control flow with a mocked/sample document. Do NOT register any cron. Verify: dry_run end-to-end on a sample target completes and routes correctly with zero real spend.

Phase 4 — harvest.budget-governor

Create harvest/budget-governor/. Track daily/monthly token/cost spend (lift the accounting pattern from discover.brain-discovery-orchestrator). On cap-reached, build the three-option decision payload and call notify.push in dry-run (do NOT actually send tonight). Budget values left unset (gated — João sets them). Verify: unit test of the cap logic; confirms it would emit action+critical at the cap; confirms it returns "paused" instead of proceeding.

Phase 5 — Semantic novelty gate + delta extraction

Implement as a function used by the worker: embed candidate (gemini-768 via LiteLLM) → k-NN cosine against nodes tagged with the term → if max cosine ≥ redundancy_threshold (start 0.92, tunable) → discard; else brain-classify extracts only the net-new fact (delta) as Markdown. Verify: unit test with two near-duplicate texts (one redundant, one with a new fact) → redundant discarded, novel delta extracted.

Phase 6 — Intelligence Object distillation (additive)

Add a distill() function producing the Markdown body + structured fields (executive_summary, new_events[], main_takeaways[], potential_consequences[], entities[], connections[], provenance {source_url, published_at, author}). Policy: do NOT persist raw body — store distilled Markdown + provenance only. Do not modify tcn.ingest-article in place; add the function and wire it for the harvest path only. Verify: distill a sample article → object has all fields, no raw body, provenance present.

Phase 7 — Adaptive cadence (pure function)

Implement next_interval(prior_interval, found_novelty, category): bounded [floor, ceiling]; category prior seeds initial; found → max(floor, interval/2), none → min(ceiling, interval*1.5). Not ML. Verify: unit tests for the Enzo-Ferrari (backs off to ceiling) and Ferrari-F1-team (stays at floor) cases.

Phase 8 — Connections → AGE + bounded enqueue

In the ingest path, for each connection: check existence in down_rabbit_hole_graph; exists → create LINKS_TO edge; not exists → harvest.enqueue-term with depth = parent.depth + 1, hard cap 3. Verify: unit test: known entity → edge; unknown → enqueued at depth+1; depth 3 → not enqueued.

Phase 9 — Admin UI tab (WRITE ONLY, do not deploy)

Write /opt/rabbithall/templates/.../curation_harvest.html (light theme, CSS vars from admin_dashboard.html) + the route in app.py, listing (a) pending harvest_targets, (b) quarantine (Six Eyes 4–6) with source URLs. Commit to the rabbithall feature/harvest-loop branch. Do NOT rh-deploy, do NOT restart rabbithall. Verify: template + route present in the branch; no deploy performed.

Phase 10 — Commit + report

Commit all work to feature/harvest-loop in each repo (gateway + rabbithall). Push the feature branch (not main). Write the morning report (below) to specs/brain/2026-06-20-harvest-OVERNIGHT-REPORT.md in jb-workspace and push.

3. GATED steps (morning, with João — do NOT do tonight)

  1. Apply any non-additive schema change.
  2. Set budget values in harvest.budget-governor.
  3. Register the crons (worker cadence).
  4. Run the first real (paid) harvest.
  5. Deploy the /admin/curation/harvest tab to the live hub (rh-deploy).
  6. Merge feature/harvest-loop → main after review.

4. Morning report template (fill and push)

# Harvest Loop — Overnight Report (2026-06-21)
## Done (per phase): 0 …10 — status, what was created, verification result
## Failed / skipped: phase, reason, what is needed
## Branch: feature/harvest-loop @ <sha> in <repos>
## DB: additive migration applied? (table/columns) + down script path
## GATED — awaiting João: budget values, cron schedule, first harvest, admin deploy, merge to main
## Cost incurred tonight: <should be ~0; list any LLM calls>
## Recommended next action for João

5. If you get stuck

If a phase is ambiguous or a verification fails and you cannot resolve it safely, STOP that phase, record the exact blocker in the report, and move to the next independent phase. Never guess on a production-affecting action. Leaving safe, partial, well-documented progress is the correct outcome — finishing everything is not required.