Knowledge Harvest Loop — Design Spec¶

Date: 2026-06-20
Status: Design approved (pending written-spec review)
Owner: João Luís Brazão
Scope: Down a Rabbit Hole brain (down_rabbit_hole DB) + jb-skills-gateway, reusable across all knowledge domains
Version prefix (proposed): harvest-v

1. Purpose¶

A self-expanding, self-pacing service that turns discoveries into distilled intelligence and keeps the second brain perpetually current — at a supportable cost — without ever storing raw source material.

The brain is a production engine, not an archive. It exists to create new posts and analysis. Therefore it stores the distillate (the "sumo"), never the photocopy.

Design principles (non-negotiable)¶

Quality over quantity. Store only intelligence that justifies being shown: genuinely new events, key takeaways, potential consequences, entities, and connections. Trivia (born/died dates, conference intros, parallel chatter) is noise and is discarded.
No raw storage. The original article/transcript/PDF is never persisted. Only the source URL + published date + author are kept as provenance.
Anti-plagiarism: with no source text to copy, output is transformative by construction; copying would require reverse-engineering.
Transparency / defensibility: the URL lets João attribute and prove sourcing under challenge ("I did not invent this — here is the original, verify it").
MD/CSV I/O contract. Everything in and out of pgvector/Postgres is Markdown (prose/distillate, the embedded body) or CSV/tabular columns (structured: entities, connections, scores, cadence). Sources are normalized to clean Markdown at the door so formatting noise never enters embeddings. Images/media deferred (store reference + text caption, never the binary).
Catalog-first / sovereignty. Reuse existing components; build only the thin glue. No new external tools (respects the 12-month moratorium). New code is first-party FastAPI/skills only.
Bounded autonomy. Self-expansion is real but governed: depth cap, per-term adaptive cadence, and a daily/monthly budget governor that pauses and asks João via Telegram. Human-in-the-loop for anything uncertain.

2. The two problems¶

This service solves two distinct problems that share one spine and feed each other.

	P1 — Term Harvest (new ontology terms)	P2 — Freshness Watch (existing terms)
Trigger	a genuinely-new term is born (not in `taxonomy_terms`)	an existing term is "stale" per its adaptive cadence
Key question	"what is this new concept?" (depth)	"did anything genuinely new appear since last time?" (freshness)
Dedup level	term-level (does the term exist?)	fact-level / semantic (does this add a fact I lack?)
Goal	learn everything about the new concept	keep the existing current; ingest only the delta

Cross-feed (the virtuous loop): - P2 → P1: watching an existing term surfaces content → a connection/entity not yet in the brain → enqueued as a P1 target. - P1 → P2: a new term, once ingested, becomes "existing" → enters P2's adaptive rotation.

Both share the back half: Six Eyes → auto-ingest (≥6) / discard (<3) / quarantine (3–6) → content-curator → pgvector + AGE auto-link → ontology-enricher.

3. Architecture overview¶

P1 (new term)                         P2 (freshness watch — existing terms)
  harvest_targets (queue)                stale-first rotation (search_topology.last_seen)
       │                                      │  + freshness filter (published_at > last_seen)
       │                                      ▼
       │                                 SEMANTIC NOVELTY GATE (pgvector)  ──► redundant? DISCARD
       │                                      │  novel? extract delta only
       ▼                                      ▼
       └────────►  Deeper Research  ──►  normalize→Markdown ──► DISTILL (Intelligence Object)
                                                │
                                                ▼
                                          SIX EYES  ──► <3 DISCARD
                                                │        3–6 QUARANTINE → /admin/curation/harvest (with URLs)
                                                ▼ ≥6
                                          content-curator → nodes (MD body) + pgvector + AGE auto-link
                                                ▼
                                          ontology-enricher → new entity/connection?
                                                │   exists in AGE? → LINKS_TO edge
                                                └─ not exists → harvest_targets (depth ≤ 3)   ⟲

  harvest.budget-governor: cap reached → notify.push (action+critical)
       └─► Telegram: "Budget exhausted. [Wait next month] · [Raise cap] · [Accelerate to clear queue: N]"

The whole thing is a domain-agnostic service. Any VPS app (TCN, geopolitics, AI, CateringAssiste, Gamine) invokes it with an envelope (term, topology, domain, destination); only those four vary.

4. Components¶

Legend: [reuse] exists today · [extend] modify existing · [new] build.

4.1 `harvest_targets` table [new]¶

The research-target queue (distinct from discovery_queue, which holds content candidates).

Column	Type	Notes
`id`	uuid PK
`term`	text	the ontology term to research
`topology`	text	category/lens (e.g. `Iconic Designers`)
`domain`	varchar	`tcn`, `geopolitics`, `cateringassiste`, …
`destination`	jsonb	`{brain, curation_panel, ingest_target}`
`status`	varchar	`pending` → `researching` → `done` / `discarded` / `quarantined`
`depth`	int	0 for direct; ≤3 for connection-trail descendants
`parent_target_id`	uuid	provenance for trail-following
`priority`	int	seeds worker ordering
`enqueued_at` / `started_at` / `finished_at`	timestamptz
`tokens_spent`	int	budget accounting
`result_summary`	text	what happened (audit)

Standard fields: created_at, updated_at, deleted_at. Unique on (term, domain) where status IN ('pending','researching') to prevent duplicate enqueues.

4.2 `harvest.enqueue-term` [new] — the public API¶

The single entry point. Accepts the envelope {term, topology, domain, destination, depth?, parent_target_id?}. Dedups against harvest_targets + checks the term doesn't already exist in taxonomy_terms (for P1). Idempotent. Callable by Hermes, by ontology-enricher, or by any app.

4.3 `harvest.worker` [new] — paced executor (cron)¶

Runs on cron (e.g. every N minutes). Per cycle: 1. Check budget via harvest.budget-governor; if exhausted, stop and let the governor notify. 2. Pull up to K targets ORDER BY priority, depth ASC, enqueued_at. 3. For each: call Deeper Research (dr_orchestrator:8200), with polite pacing (sleep between sources, rate-limit respect — never crawler-like). 4. Pipe results through normalize → distill → (P2: novelty gate) → Six Eyes → ingest. 5. Update last_seen / last_novelty_at, recompute cadence, account tokens.

4.4 `harvest.budget-governor` [new] — interactive cap¶

Tracks daily/monthly spend (lifts the budget logic already in discover.brain-discovery-orchestrator, which currently skips silently). When the cap is reached it does not skip silently — it fires notify.push (category=action, severity=critical) → Telegram with three actions: - Wait for next month's budget, - Raise the cap now, - Accelerate (temporarily raise to clear N queued targets).

Hermes interprets the reply and resumes/raises/holds.

4.5 Semantic novelty gate + delta extraction [new] (P2 core)¶

The genuinely-new technical piece. The pgvector substrate already exists — content-curator already computes MAX(1 - (embedding_768 <=> v)) against existing nodes. We add the decision + delta: 1. Embed the candidate (gemini-embedding-001, 768-dim). 2. k-NN against existing nodes tagged with that term. 3. If max cosine ≥ redundancy_threshold → discard before spending Six Eyes. 4. If novel → cheap LLM step (brain-classify tier) compares against top-k and extracts only the net-new fact (delta) as Markdown; the delta is what proceeds to Six Eyes + ingestion, auto-linked to the node it augments.

⚠️ brain.novelty-detector does not do this. That skill detects publication-worthiness (cross-cluster bridge / contrarian shock / entity relink → idea_candidates). It is editorial novelty, not content dedup. They are separate concerns.

4.6 Intelligence Object + ingestion policy [extend]¶

Replaces the "summary + 5 takeaways, plus raw body" behavior of tcn.ingest-article. The distilled object (Markdown body + structured columns):

executive_summary       : 2–4 dense sentences (what changed / why it matters)   [MD body]
new_events[]            : only genuinely-new events/facts                         [MD list]
main_takeaways[]        : actionable insights                                     [nodes.main_takeaways]
potential_consequences[]: implications / "so what?"                               [new column / metadata]
entities[]              : people, brands, circuits, teams                         [CSV/columns + AGE]
connections[]           : { entity, relation, already_in_brain: bool }            [new column / metadata + AGE]
provenance              : { source_url, published_at, author }                    [columns]

Storage policy change: body_raw / body_plain are no longer persisted. The node body is the distilled Markdown; only provenance is kept. new_events, potential_consequences, and connections are new fields (own columns or metadata jsonb).

4.7 Normalization at the door [new]¶

Before distillation, every source format (HTML, YouTube transcript, PDF, PPTX) is converted to clean Markdown — stripping layout noise, intros, and parallel chatter — so embeddings reflect content, not formatting. Images/media: store reference + text caption only (deferred).

4.8 Connections → AGE graph + bounded enqueue [extend]¶

Each connection is checked against the AGE graph (down_rabbit_hole_graph: Entities, LINKS_TO, MENTIONS — already seeded): - exists → create LINKS_TO edge (problem already handled), - not exists → harvest.enqueue-term with depth = parent.depth + 1 (hard cap 3) and parent_target_id for provenance.

The ingested distillate is therefore itself a node of live links, maximizing pgvector + graph discovery.

4.9 Adaptive cadence [new] — bounded backoff, not ML¶

Per term, fully explainable and tunable: - check_interval (days) bounded [floor, ceiling]. - Category prior seeds the initial interval: active team/championship → ~1 day; annual event → calendar-linked (monthly off-season); deceased/historical person → frequent during initial backfill, then long. - After each check: novelty ingested → interval = max(floor, interval / 2); nothing new → interval = min(ceiling, interval × 1.5). - State: reuse search_topology.last_seen as last_searched_at; add last_novelty_at + check_interval.

Worked example — Enzo Ferrari (deceased founder): after the backfill captures ~99%, interval climbs to ceiling (e.g. quarterly) but never stops, catching the rare new historical discovery. Ferrari F1 team: stays at the floor (daily) because it keeps yielding novelty. The cost/benefit curve emerges from the algorithm, not from manual tuning.

4.10 Freshness filter [extend]¶

harvest.worker / discovery queries add a date filter (Brave freshness param + capture published_at) so only content published after the term's last_searched_at is considered. Directly implements "8-day-old news, last search 15 days ago → must pick it up."

4.11 HITL panel `/admin/curation/harvest` [extend]¶

New tab in the existing /admin/curation hub (light theme, colorblind-safe per house rules). Two lists: - Pending terms awaiting research (from harvest_targets), - Quarantine (Six Eyes 4–6) with source URLs + access to sources so João decides continue / ingest / discard.

4.12 Six Eyes thresholds [reuse]¶

<3 discard · 3–6 quarantine (human) · ≥6 auto-ingest. Already implemented in tcn.six-eyes-scorer / discover.six-eyes-scorer. No change.

5. Reuse vs. build (grounded in current code)¶

Capability	Today	Action
Content discovery for known terms	`tcn.taxonomy-discovery` (rotates 285 terms, dedups articles into `brain_staging`)	[reuse]/[extend] add freshness
Domain-agnostic candidate queue	`discovery_queue` (+ findings/rejected/feedback), currently empty	[reuse] for content; [new] `harvest_targets` for terms
Budget governor	`discover.brain-discovery-orchestrator` (monthly cap, skips silently)	[extend] make interactive via `notify.push`
Six Eyes scoring	`tcn.six-eyes-scorer`, `discover.six-eyes-scorer`, `brain.six-eyes-triage`	[reuse]
Deep research engine	Deeper Research (`dr_orchestrator:8200`) + `tcn.deep-research`	[reuse]
Ingestion + summary/takeaways	`tcn.ingest-article` (stores raw + summary + 5 takeaways)	[extend] distill-only, drop raw
Semantic similarity	`content-curator` cosine `<=>` against `embedding_768`	[extend] add gate decision + delta
New-term proposal	`brain.ontology-enricher`, `tcn.taxonomy-extractor`	[extend] route new terms to `harvest_targets`
Graph	AGE `down_rabbit_hole_graph` (Entities, LINKS_TO, MENTIONS)	[reuse]
Per-term last-search	`search_topology.last_seen` (already ordered ASC)	[extend] add `last_novelty_at`, `check_interval`
Notifications	`notify.push` (action+critical → Telegram)	[reuse]
HITL panel	`/admin/curation` hub	[extend] add Harvest tab

Net new: harvest_targets table; skills harvest.enqueue-term, harvest.worker, harvest.budget-governor, semantic novelty gate; Intelligence Object schema + normalization; adaptive-cadence columns; curation tab. Everything else is reuse/extend.

6. Convergence & cost controls¶

Depth cap (≤3) on connection-trail recursion.
Adaptive cadence prevents over-polling stable terms.
Budget governor is the hard ceiling; pauses and asks rather than overspending.
New terms always queue (pending); only the worker triggers research — never ontology-enricher directly. Guarantees pacing and a single throttle point.
Term-level + fact-level dedup prevents re-ingesting known content.

7. Open / deferred¶

Image & media handling — deferred (reference + caption only for now).
Exact thresholds (redundancy_threshold, cadence floor/ceiling, category priors) — to be tuned during implementation against real TCN data. Six Eyes cutoffs fixed at 3/6.
harvest_targets location — down_rabbit_hole public schema (default); revisit if multi-DB destinations (e.g. CateringAssiste/Gamine) need their own.
MinIO cold archive of raw — rejected (João: no value, adds risk/noise). URL-only provenance stands.
Worker runtime — cron-driven gateway skill (default); revisit a long-running worker only if pacing granularity proves insufficient.

8. Six Eyes review of this design¶

Attention: the triggering notification reported new articles, not new taxonomy terms — the spec corrects that framing and addresses the real gap (no research fires on genuinely-new terms today).
Emotion: the driver is editorial integrity (anti-plagiarism, defensibility) as much as enrichment.
Framing: two problems, not one; conflating them would have produced a wrong design.
Structure: one spine, two fronts, explicit cross-feed and convergence controls.
Contrast: rejected a dedicated microservice (B) and overloading the existing orchestrator (C) in favor of gateway skills + reuse (A).
Integrity: reuse/build claims are grounded in the live code (content-curator cosine, ingest-article raw+summary, novelty-detector ≠ content dedup, search_topology.last_seen).

9. Phasing (for the implementation plan)¶

Indicative only — detailed plan to be produced by the planning step.

Spine + P1 (TCN): harvest_targets, enqueue-term, worker, budget-governor, distill-only ingestion, curation tab.
P2 (TCN): freshness filter, semantic novelty gate + delta, adaptive cadence.
Connections: AGE existence-check + depth-bounded trail.
Generalize: envelope for geopolitics / CateringAssiste / Gamine.