Skip to content

VPS Full Infrastructure Audit — srv1345342 (76.13.44.83)

Date: 2026-06-10 Scope: Hostinger control panel + full VPS (containers, nginx, databases, cron, /opt, admin screens, skills gateway, security, system health, DNS) Method: 9 parallel read-only audit agents over SSH + 22 adversarial verifications of every CRITICAL/HIGH finding + Hostinger hPanel review via browser. No state was modified. Every finding below survived independent re-verification; severities shown are post-verification.

1. Executive Summary

Overall status: YELLOW — the production surface is healthy (all 41 running containers healthy, all public sites up, disk/RAM/CPU comfortable), but the safety layer underneath it is broken: backups have silently produced nothing for 12 days, the restore-test that should have caught it has never worked, there is no off-site copy, and several TLS certificates are failing renewal with hard expiry dates in 17–22 days.

The 5 things that matter most (in order)

# Finding Severity Fix effort
1 Daily backups dead since May 22 — /opt/backup.sh lost its execute bit in the May 21 edit; 20 nightly Permission denied failures; newest backup of all 10 production DBs is 2026-05-29 (manual run) CRITICAL chmod 755 — one command
2 No off-site backup has ever existed — rclone never configured; the only copy of everything (10 DBs incl. Vaultwarden + Forgejo volumes) sits on the same physical disk as production CRITICAL ~1h (rclone config + cron)
3 Backup restore-test broken since inception — set -euo pipefail + zcat \| head SIGPIPE aborts the script after the first log line, every week since April; it also has no staleness check and no alerting. The safety net was decorative. CRITICAL ~30 min
4 brain.joaoluisbrazao.com cert expires 2026-07-02 and renewal fails on every run — server-level return 301 in brain.conf bypasses the ACME location; same family of breakage will block the Chatwoot cert renewal in mid-July (parse-broken renewal conf) HIGH ~30 min
5 Second Brain API is the default vhost, open to the world without authentication — any request to the raw IP https://76.13.44.83/ or to any unmatched subdomain serves the full Brain Graph Explorer; /api/v1/stats and /api/v1/graph/explore return data with no auth (17,683 nodes incl. 1,357 personal notes). Confirmed empirically from outside. HIGH (decision) ~15 min for a default_server 444 catch-all

Health scoreboard by area

Area Status Headline
Containers / Docker 🟢 GREEN 41/41 running healthy, zero crash loops, log rotation OK, disk fine
Nginx routing / TLS 🟡 YELLOW All live vhosts work; 5 cert lineages failing renewal, 2 renewal configs parse-broken, default-vhost exposure
Databases 🟢 GREEN Both PG instances healthy, AGE+pgvector OK, no unknown DBs; hygiene items only
Cron jobs 🔴 RED 17 of ~45 active jobs broken or never-succeeded, 3 exact duplicates
/opt directories 🟡 YELLOW ~12.3 GB removable; obsolete dangerous compose file; dev/prod inversion
Admin screens 🟡 YELLOW 6 hanging screens, ~40 .bak templates, 3 dark-scheme violations, BYPASS_TOKEN active
Skills gateway 🟡 YELLOW 85 loaded, core pipelines work; 2 fail to load (perms), 9 cron-wired skills failing, ~26 with no caller
Security 🟡 YELLOW UFW+fail2ban good; Docker bypasses UFW on 3 ports; SSH password-auth latent; pending kernel reboot
System health 🟡 YELLOW Resources excellent; logrotate broken since Jun 05; 4 failed systemd units; reboot pending

Recurring root-cause patterns (worth fixing once, structurally)

  1. The 640-permission pattern strikes again — ssh-edit-block/docker cp edits drop the execute/read bits. It killed /opt/backup.sh (CRITICAL #1) and brain.content-curator and ingest.podcast-processor. The documented ACL fix (feedback_acl_permanent_fix) was applied to the web root but never to /opt scripts or the skills directory.
  2. Silent failure everywhere — every broken cron logs to a file nobody watches. 17 jobs failed for days/weeks with zero alerts. The gateway even records failed skill runs as "success" in command_runs.
  3. Duplicate scheduling layers — root crontab vs /etc/cron.d define the same jobs 3×; host nginx/certbot vs containerized nginx/certbot fight over the same responsibilities and generate permanent error noise that buried the real cert failures.

2. Hostinger Control Panel

Item Value Assessment
Plan KVM 8 — 8 vCPU, 32 GB RAM, 400 GB disk, 32 TB bandwidth Right-sized; usage is 7% CPU, 23% RAM, 84/400 GB disk, 0.019/32 TB
VPS expiry 2027-04-11, auto-renew ON ✅ OK
Location / OS Paris — Ubuntu 24.04 LTS ✅ OK
Panel firewall 0 rules ⚠️ Everything depends on the VPS's own UFW — which Docker bypasses (see §4). Consider adding Hostinger-level rules mirroring 22/80/443/2222/25 as a second layer.
Backups Daily schedule, 4 snapshots/backups present ✅ Exists — but provider snapshots are an untested last resort, not a substitute for the broken pg_dump pipeline
Malware scanner Not installed Acceptable — ClamAV + Falco already run on the VPS itself
Domains joaoluisbrazao.com (exp 2027-02-13), joaoluisbrazao.cloud (exp 2027-02-07), both auto-renew ON ✅ OK. gamine.org is registered elsewhere (SiteGround), correctly NOT on this VPS

DNS zones (Hostinger nameservers)

joaoluisbrazao.com — A records all → 76.13.44.83 unless noted: @, admin, brain (⚠️ has BOTH an A record and a self-referencing CNAME brain → brain.joaoluisbrazao.com — invalid pair, delete the CNAME), tcn, git (Forgejo ✅), tasks (Vikunja ✅), headscale ✅, chat (⚠️ no vhost — orphan), npm (⚠️ orphan), auto (⚠️ orphan), llm (⚠️ orphan — and pointing the name "llm" at the box advertises the AI proxy), gestso-sandbox (⚠️ orphan — sandbox removed 2026-05-13), ftp → 82.29.199.237 (⚠️ foreign IP, old shared hosting — orphan), www → Hostinger CDN CNAME. MX/SPF/DKIM/autodiscover → Hostinger mail. DMARC is p=none (consider quarantine later).

joaoluisbrazao.cloud: @, www, vault ✅, chat (Chatwoot ✅), uptime (⚠️ no vhost — Uptime Kuma is actually at /uptime/ on admin; orphan record), app (⚠️ orphan), mail (⚠️ no vhost; mail is Hostinger MX — orphan unless kept for SMTP intake branding). Same Hostinger mail records.

Missing records for declared intentions: admin.joaoluisbrazao.cloud (#714), docs.joaoluisbrazao.com (#565), research.joaoluisbrazao.com (#906) — all confirmed absent; the corresponding nginx vhosts/plans are waiting on them.

Why orphan DNS records matter here: every unmatched name resolves to the VPS and gets served by the default vhost = the Second Brain app (finding #5). Until the catch-all is fixed, each stray record is another door to the brain UI.

3. What is CORRECT (verified working as intended)

Platform. All 41 running containers healthy, zero crash loops (max RestartCount=1), every container has a memory limit and unless-stopped policy. Docker daemon has 10m×3 log rotation + live-restore. Disk 22% used (303 GB free), inodes 3%, RAM 24 Gi available, swap untouched, load 0.31 on 8 cores. Daily docker image prune cron keeps images at only 699 MB reclaimable.

Routing & TLS (the working part). nginx -t clean; all 13 public domains with DNS return 2xx/3xx; every proxy_pass upstream maps to a running healthy container; rate-limiting active on /admin/login; the resolver-variable pattern protects nginx from rabbithall recreations; SMTP stream proxy to brain-mail-intake works. The CLAUDE.md §12 "SSL .com + .cloud expire 2026-06-28 URGENTE" item is OUTDATED — both main certs (and admin, tcn, vault, git, headscale, tasks-0001) renewed to 2026-08-27. The #574 docs-exposure fix is confirmed live (joaoluisbrazao.com/docs/ → 404).

Databases. AGE 1.5.0 + pgvector 0.8.2 coexisting correctly in down_rabbit_hole; graph down_rabbit_hole_graph present. Every DB maps to a known consumer — no unknown/test DBs; dropped geopolitics confirmed gone. Neither PG instance is network-exposed. Connection counts modest; autovacuum current. Deeper Research (10 tables) and Expense Report (er schema, Alembic) exactly as documented. Per-app roles exist for chatwoot/er/jb2b/gamine_brain/jb_brain/postiz.

Working pipelines. brain-sync (every 10 min, 31×200 OK in 6h), brain-digest (posted today), brain-cluster (118 clusters), brain-analyze (9 insights), the full TCN chain (discovery → Six Eyes scoring 7.2–8.2 → auto-approve → ingest-article/video, 220 items fetched today), audit.orchestrator weekly (246 findings, Telegram delivered), search.topology_sync, Six Eyes browser extension scoring (6×200 today), jbsb.memory-decay (21,836 links), gamine-angle-generator, taxonomy-realign (flock-protected). Rabbithall runs from the correct /opt/rabbithall-compose/. Forgejo is canonical VCS with 19 repos. Uptime Kuma alive and writing.

Security (the working part). UFW default-deny with minimal allowlist (22/80/443/2222/25); fail2ban active 4 weeks, 163 IPs banned; root SSH key-only, no non-root account has a password or authorized_keys; unattended-upgrades on; Falco + falcosidekick + ClamAV running; secrets in active compose files all ${VAR}-interpolated; admin screens consistently @admin_required.

4. What is WRONG (all findings verified; post-verification severity)

CRITICAL — backup chain (three compounding failures)

C1. Nightly backups dead since 2026-05-22. /opt/backup.sh is -rw-r----- (execute bit lost in the May 21 17:59 edit — the known ssh-edit-block pattern). Cron 0 3 * * * /opt/backup.sh fails nightly: 20 consecutive Permission denied entries (8 before the manual May 29 run, 12 after). Newest dumps for all 10 databases (litellm, gamine, vikunja, ghrepos, dashboard, down_rabbit_hole, gamine_brain, darah, chatwoot, jb2b) and 4 volumes: 2026-05-29. → Fix: chmod 755 /opt/backup.sh, run once manually, verify. Add expense_report DB to the dump list (it has never been backed up). Add a staleness alert.

C2. No off-site copy has ever existed. rclone listremotes is empty; /etc/rclone/offsite.conf was never created (only the .template exists); the offsite script ran once on May 29 and errored. The 13 GB at /var/backups/eureekka is the only copy, on the same /dev/sda1 as production — including the Forgejo (sovereignty git host) and Vaultwarden volumes. Hostinger's 4 daily snapshots are the only external mitigant, and they're untested. → Fix: configure the rclone remote from the existing template, schedule offsite-backup.sh after the 03:00 local run, verify first sync.

C3. The restore-test never worked. /opt/backup-restore-test.sh (Sundays 04:00) has logged only START lines since April — set -euo pipefail + zcat | head -1 SIGPIPE aborts it before any verdict. Even fixed, it checks the 4 newest dumps regardless of age and alerts nobody. → Fix: make the pipeline SIGPIPE-safe, add a "newest backup older than 26h = FAIL" check, wire FAIL to Telegram.

HIGH

H1. brain.joaoluisbrazao.com cert renewal failing; expires 2026-07-02 (22 days). Server-level return 301 in brain.conf:80 redirects ACME probes; the followed HTTPS request hits the 443 block which proxies everything to jb-brain-api → 404. Failing twice daily. This is #887, previously triaged NICE — it must be re-prioritized. Working vhosts (tcn.conf) wrap the redirect in location / { } — copy that pattern, reload, renew.

H2. Chatwoot cert will fail renewal mid-July. /etc/letsencrypt/renewal/chat.joaoluisbrazao.cloud.conf was generated with host-filesystem paths and certbot skips it as invalid every cycle (expected .../cert.pem to be a symlink). The live cert is valid to 2026-08-14 but has no renewal path. Re-issue inside infra-certbot with container paths before mid-July.

H3. Second Brain served as default vhost without auth (decision required). brain.conf is alphabetically first, so it is the de-facto default_server for :80 and :443. Confirmed from outside: https://76.13.44.83/ and every unconfigured subdomain (npm., auto., llm., mail., app., …) 307-redirect into the Brain Graph Explorer, and /api/v1/stats + /api/v1/graph/explore return full data with no authentication — 17,683 nodes including 1,357 personal notes. If brain.joaoluisbrazao.com is meant to be a public showcase, that is a product decision — but the raw-IP/catch-all exposure is not. → Add an explicit default_server block returning 444; then decide whether the brain API itself needs an auth gate for non-public node types.

H4. Skills gateway: 2 skills fail to load (640 perms) — TCN curation + podcast pipelines dead. brain/content-curator/main.py and ingest/podcast-processor/main.py are -rw-r----- root:root on the bind mount; the gateway (uid 999) gets PermissionError at load. The content-curator cron has 404'd every 4h for ~10 days (this is the #847-fixed skill, broken again by the same pattern). → chmod 644 both + reload; apply a default POSIX ACL on /opt/jb-skills-gateway/skills so this never recurs (same fix as the web root).

H5. JBSB Intelligence System: all 4 engines + telegram-notify return 500 on every run — root cause found. The gateway's run_skill calls skill["run"](body) without awaiting async functions; all 5 failing jbsb skills are async def run, so FastAPI tries to serialize a coroutine → 500. The system "LIVE 2026-05-31" has never delivered an editorial provocation. Only the sync jbsb.memory-decay works. → One fix in /app/gateway/main.py (if inspect.iscoroutine(result): result = await result) revives all five.

H6. ingest.podcast-orchestrator: doubled path bug — 116 failures, 0 successes. Builds /app/skills/ingest/ingest/podcast-monitor/main.py (duplicated ingest/). The podcast pipeline (#888, #494-497) has never functioned. → Fix the path join + the H4 chmod on podcast-processor.

H7. brain.graph-rebuild failing nightly — dollar-quote breakout = injection vector. Node content containing $$ breaks out of the dollar-quoted Cypher body (syntax error at or near "BORING"); 12 consecutive nightly failures, so MENTIONS/LINKS_TO edges are never rebuilt. Because arbitrary ingested content reaches query context, this is also a SQL/Cypher injection surface. → Parameterize or use unique dollar-quote tags.

H8. ghrepos catalog scanner crashing every 3h — KeyError: DB_PASSWORD (env no longer exported by run_scanner.sh). The catalog behind the CATALOG-FIRST rule is not updating. → Restore env sourcing.

H9. World-writable root cron scripts + plaintext LiteLLM master key. vps-health-check.sh and vps-daily-report.sh are mode 777 and run as root (hourly/daily) — local-account → root escalation path; the daily report embeds the LiteLLM master key inline. → chmod 750 root:root both; move the key into /opt/infrastructure/.env (600).

H10. Docker bypasses UFW — 3 app ports internet-exposed. DOCKER-USER chain is empty; confirmed reachable from the public internet: :3001 Uptime Kuma login UI, :8140 gamine-brain-api (raw FastAPI with /api/v1/auth/login, 2FA endpoints, open /docs + /openapi.json, plain HTTP, no rate limiting), :8141 gamine-brain-web. (headscale :3478/udp is by-design STUN — fine.) → Rebind all three to 127.0.0.1: in compose and route through nginx, or add DOCKER-USER drop rules. Remember the Hostinger panel firewall has 0 rules — nothing upstream saves you.

MEDIUM

# Finding Note
M1 litellm OOM-killed 26× Jun 05–09 (2 GiB cgroup), auto-restarts in ~1s Load-correlated with the Second Brain bulk import, quiescent since Jun 09 — but unmitigated; next bulk job re-triggers it. Raise to 3–4 GB (24 Gi free) + add Kuma monitor
M2 SSH PasswordAuthentication effectively ON (50-cloud-init.conf wins over the later no) Currently nil exploitability (no passwords set, fail2ban active, 163 bans) but latent: one passwd for a family account makes it brute-forceable. Set no in 00-hardening.conf
M3 Pending kernel reboot (6.8.0-117 installed vs -111 running, 34 days) Also the likely cure for M4. Pre-reboot: disable host nginx.service (will race infra-nginx for :80/443 at boot)
M4 logrotate broken since Jun 05 (systemd NAMESPACE error; syslog 172 MB growing ~17 MB/day) Same signature on man-db.service; characteristic of kernel/systemd awaiting reboot
M5 Host certbot (timer + cron.d) fails every 12h forever — manages an admin cert nginx doesn't use; noise buried the real failures Disable host certbot; infra-certbot container is the single source of truth
M6 Obsolete /opt/rabbithall/docker-compose.yml still present — embeds retired admin master-password hash + 6 family emails; running it would take down the hub and re-introduce the removed master-password bypass Rename to .OBSOLETE-DO-NOT-USE + README pointer
M7 Vaultwarden WebSocket push broken — /notifications/hub proxies to :3012 which the container doesn't expose (502); clients silently poll Point at :80 or delete the location (modern Vaultwarden serves WS inline)
M8 LiteLLM_SpendLogs = 819 MB (98% of litellm DB), unbounded Enable retention / 90-day purge; daily aggregates preserve reporting
M9 chunks table 2,130 MB — two dead 1536-dim embedding columns double its size (all embeddings are 768-dim since Ollama removal) Plan column drop + pg_repack: ~1 GB reclaim
M10 DB password hygiene — litellm bootstrap-superuser password shared across 6+ containers, readable via docker inspect; weak human password on jb_brain_user Rotate; create scoped roles for deeper-research/dashboard/vikunja
M11 Cron duplicates — weekly audit 2× (Sun 23:00), monthly audit line dead (unescaped %), architect.reporter 2× (double LLM spend) Delete the 3 root-crontab lines; keep /etc/cron.d as single source
M12 Dead/broken crons: geopolitics refresh (DB dropped), discover.content-nuggets + brain-score-batch (renamed DB jb_brain), check-deepseek-balance (source under dash), gamine_angle_cron (hardcoded stale container IP), brain.publication-brief (enum cast), brain-lint (timeouts ×4) Each is a one-line-ish fix; list in §8
M13 jb-brain dev/prod inversion — production runs from /opt/dev/jb-brain, stale decoy clone in /opt/prod/jb-brain Migrate or delete the decoy + document
M14 Old infra-docs + backups physically inside the public TCN web root (/opt/downrabbithole/html/docs/, _backup/, *.bak) One nginx regression away from re-exposure (#574 history). Delete from web root
M15 Duplicate uptime-kuma compose in /opt/monitoring — would collide by container name if ever up'd Delete
M16 Internet-exposed custom SMTP intake (port 25, aiosmtpd) — intentional, but relay/allowlist/rate-limit posture unverified Verify recipient allowlist + fail2ban coverage
M17 rabbithall /admin/auto-login?token= bypass route is ACTIVE (BYPASS_TOKEN set in container env) — contradicts the 2026-06-10 "no emergency bypass" decision Unset BYPASS_TOKEN / remove route, or re-document the decision
M18 3 dark-background screens live (discovery.html, quarantine.html --bg:#0f172a; brain_landing.html #0d0d0d) — violates the non-negotiable light-scheme rule and two ARE linked from brain_landing Re-skin to #f0f2f5 canonical scheme

LOW (selection)

13 containers without healthchecks (incl. er-celery with healthcheck explicitly NONE); 9 containers without CPU limits (the entire deeper-research stack — the CPU-heaviest workload); 9 images on :latest/rolling tags; gamine-db running on an untagged image (prune/recreate hazard — retag before any prune); gamine-web image 7.61 GB (22% of all image storage, needs multi-stage build); headscale 57 MB uncapped live log (predates daemon log-opts — recreate); /api/review/ reachable over plain HTTP on the :80 vhost; headscale.conf has backslash-escaped nginx variables (malformed WebSocket Upgrade header); gamine.conf actually serves joaoluisbrazao.cloud (rename for incident-response sanity); unused zone=api rate-limit; Vikunja env still carries a sqlite path; root-only SSH access pattern; sensitive Vaultwarden DB copies parked in /tmp (vw*.db from the Jun 09/10 login fix — delete); /tmp crontab backup of the suspended Apify lines won't survive the pending reboot — move it; volume backups for gamine media (157-byte tar vs 5,655 files) and vikunja files effectively empty (#59 confirmed real — wrong volume names in backup.sh); DMARC p=none.

5. Orphans — installed but connected to nothing

5a. Files & directories (~12.3 GB reclaimable; disk is only 22% used, so this is hygiene, not urgency)

Item Size Verdict
/root/vps-backup-2026-03-29.tar.gz 5.0 GB 73-day-old full-server tarball. Move off-site or delete
Instagram export ×3 (/opt/gamine/web/…, /opt/dev/gamine/…, /opt/gamine/backups/instagram-backup/) ~5.4 GB total Imported into DB+media archive 2026-05-16. Keep ONLY the labelled backup copy; delete the other two (~3.6 GB) — especially the one inside the live web app dir (PII: DMs/followers)
/opt/backups/pre-age-*.sql.gz ×2 1.6 GB Duplicate pre-AGE dumps (3 min apart). Delete the one without .done; archive the other off-site after backups work again — until then it's ironically your freshest litellm copy
/opt/imports/second_brain_2026 1.3 GB One-shot import source (Jun 09). Delete after enrichment verification
/opt/kb-more 393 MB Corpus ingestion workspace, idle since Apr 15. Spot-check content is in the brain, then delete
/tmp/bear_export + misc /tmp artifacts ~245 MB One-shot staging. Delete (incl. the vw*.db Vaultwarden copies — security hygiene)
/opt/downarabbithole (typo twin of the web root) 1.9 MB Abandoned TCN prototype. Delete — dangerous near-duplicate name
/opt/prod/jb-brain 284 KB Stale decoy clone (prod runs from /opt/dev). Delete or make real
/opt/monitoring 8 KB Duplicate uptime-kuma compose. Delete
Empty dirs: /opt/codewiki, /opt/email-triage, /opt/gamine-cms, /opt/__pycache__ <30 KB Delete all four
/opt/gamine/backups March dumps + dangling symlink ~30 MB Dead local rotation; remove once main backup verified
/opt/ingest, /opt/infra, /opt/obsidian_brain, /opt/docs ~27 MB Review-then-delete. ⚠️ /opt/docs/vaultwarden-password-inventory.md — check content sensitivity first
/opt/git-repos/deeper-research.git ~800 KB Superseded by Forgejo remote (Jun 09). Verify remotes, then delete. KEEP geo-ingest.git — only remote until #875

5b. Docker objects

  • Networks (6 empty): gamine_default, net-gamine, net-jbsb, net-medvault, net-shared, net-tcn — remnants of the pre-infra-net scheme. docker network rm after a compose-file grep. (Keep postiz-net for the paused stack.)
  • Volumes (3 unattached): two anonymous (~96 MB, inspect then rm) + monitoring_uptime_data (283 KB stale predecessor). KEEP audit-trivy-cache (1.13 GB) — used by the weekly Trivy audit via ephemeral containers; only looks orphaned between runs.
  • Build cache: 714 MB reclaimable (docker builder prune).
  • Postiz stack: 4 exited containers + 3 DBs (postiz, temporal, temporal_visibility — 31 MB) + ES/Temporal images (multi-GB). Paused 2026-05-30 (Prisma↔AGE collision), not abandoned — but needs a decision deadline: fix Prisma search_path scoping and resume, or dump+drop the DBs, rm the containers, prune the images (#735).

5c. Certificates (infra-certbot volume)

Failing renewal every 6h and referenced by no nginx config: chat.joaoluisbrazao.com + chat.joaoluisbrazao.com-0001, gestso-sandbox.joaoluisbrazao.com, tasks.joaoluisbrazao.com (expired Apr 02, superseded by -0001), mail.joaoluisbrazao.cloud (verify brain-mail-intake doesn't load it — inspection says no mounts, so almost certainly safe). Plus the host-level admin.joaoluisbrazao.com lineage (shadowed by the container cert). → certbot delete the five + remove host certbot entirely. This clears ~all the renewal noise so real failures become visible.

5d. Cron lines (root crontab)

Dead: geopolitics refresh (DB dropped), monthly-audit guard line (unescaped %, never ran). Duplicates: weekly audit, architect.reporter (both covered by /etc/cron.d). Plus ~6 stale comment headers describing jobs that no longer exist beneath them. The 5 commented Apify lines are correctly suspended — but move the restore copy out of /tmp before rebooting.

5e. DNS records (see §2)

Delete: gestso-sandbox, ftp (.com), the self-referencing brain CNAME. Decide: npm, auto, chat, llm (.com); app, mail, uptime (.cloud) — none has a vhost; all currently leak into the brain catch-all.

5f. Skills (~26 of 88 dirs have no caller)

  • Dead legacy (remove): index.ollama-embeddings (targets removed Ollama), index.hybrid-retriever, index.pgvector-connector, discover.six-eyes-scorer + discover.threshold-calibrator (superseded — three six-eyes implementations exist; consolidate), stray top-level contact.triage dir (wrong location, diverged from live tcn.contact-triage).
  • Never wired (decide: wire or archive): self-improve.* ×3, enrich.brand-voice-analyzer, enrich.entity-taxonomy-tagger, enrich.semantic-chunker, generate.content-generator, ingest.rss-feed-ingester, ingest.document-normalizer, ingest.second-brain-injector, ingest.apify-social-scraper, ingest.youtube-monitor(-orchestrator) (#333).
  • By-design entry points awaiting wiring (KEEP): jbsb.ingest-url/voice/session-prime/session-compact/context-recall — the Telegram webhook makes no gateway calls yet (#893); publish.postiz-connector (pending Postiz decision).
  • Manual/on-demand tools (KEEP, document as manual-only): brain.brain-research, brain.pdf-extract, brain.bulk-import-local (used Jun 10), brain.ontology-validator.
  • Bug: rabbithall fires nonexistent tcn.deep-research on the curation "deep research" button — the 404 is swallowed; button silently does nothing. Implement (bridge to Deeper Research :8200) or remove the button. Also tcn.taxonomy-extractor is never actually invoked by ingest-article due to a wrong importlib path with a swallowed exception — Step 7 taxonomy extraction silently skips on every article.

5g. Database objects

postiz + temporal + temporal_visibility DBs (Postiz decision); brain_staging 5,914 rows awaiting curation (#815 + 578 geopolitics in_review — a queue for João, not a deletion).

6. Admin screens — hierarchy audit (353 routes, ~70 page templates)

Linked and healthy

Dashboard hub → commands, curation hub (14 templates share _curation_nav.html), CRM, tasks, runs/audit cockpit, knowledge-curation, search/brain quarantines, ontology cluster, domain explorer, JBSB contradictions, gamine cluster, ingestion-control. Legacy URLs 301 correctly to the unified hub.

Screen Route Recommendation
DARH island: ideas, applications, talk-to-me, audit-log (+quarantine) /admin/ideas etc. They cross-link to each other but nothing links IN. #878 is NOT done — one dashboard card fixes the whole island
JBSB editorial queue /admin/jbsb/editorial-queue Mitigant: Telegram deep-links are the designed entry — but telegram-notify is broken (H5), so right now it's unreachable in practice. Add to _curation_nav
Publication /admin/publication Link from nav or archive if the workflow is dead
TCN requests ("Talk to Us" queue) /admin/tcn-requests Add to a TCN nav group
Contacts /admin/contacts Probably superseded by /admin/crm — confirm and archive
TCN article detail /admin/tcn-article/<id> Wire links from the proposals/audit lists

Dead weight in templates/

~40 .bak/.bak-* files bind-mounted next to live templates (admin_dashboard ×8, landing ×11, ~10 .bak-curnav…) — git history on Forgejo makes them redundant; _curation_nav_new.html (staged nav never wired — finish or delete); _brain_curator_nav.html (superseded); admin_brain_curator.html + /admin/brain-curator/legacy route (retired screen kept alive); legacy discovery/quarantine/ingestion-control cluster overlaps the new curation hub and two of them are dark-themed — review for retirement.

7. Connection map (verified)

Public edge: Hostinger DNS → 76.13.44.83 → infra-nginx (:80/:443; :25 SMTP stream → brain-mail-intake:2525) → all upstreams on infra-net/app-net, loopback-only. Forgejo SSH on :2222. ⚠️ Side doors bypassing this map: :3001, :8140, :8141 (H10).

Routing: joaoluisbrazao.com → static TCN webroot (+ /api/tcn/→rabbithall, /api/darah/→darah-api) · tcn. → rabbithall · joaoluisbrazao.cloud → rabbithall + 16 locations (vikunja, ghrepos, dashboard-refresh, skills-gateway, gamine-web, er-backend/frontend, audit-reports, uptime) · admin.com → rabbithall + 13 API locations (litellm, gamine-brain-api, uptime-kuma…) · brain. → jb-brain-api (+ accidental default vhost for everything else) · chat.cloud → chatwoot · vault.cloud → vaultwarden (WS sublocation broken, M7) · git/tasks/headscale → forgejo/vikunja/headscale.

Data: infra-postgres (14 DBs, 5.1 GB) serves 9+ apps — consumer map fully verified, no unknowns; gamine-db (20 MB) serves gamine-web only. Redis → litellm/chatwoot/er. MinIO ×2 (gamine, er). All AI traffic → litellm:4000 (loopback) → providers.

Automation: ~45 active cron jobs → 28 hit the skills gateway (:8095, 85 skills loaded); 18 verified producing fresh ok:true results; 17 broken (§4); 3 duplicated. Systemd: jb-heal-daemon + deploy-webhook are live units (not orphans); host nginx/certbot/logrotate/man-db are the 4 failed units (2 orphan-by-design, 2 reboot-fixable).

8. Cleanup & remediation plan

Phase 0 — Today (≈1h total, trivial one-line commands; recommendation: do immediately)

  1. chmod 755 /opt/backup.sh → run manually → verify dumps land; add expense_report to dump list.
  2. chmod 644 .../brain/content-curator/main.py .../ingest/podcast-processor/main.py → reload gateway. Add default POSIX ACL on the skills dir (and /opt scripts) so the 640 pattern can't bite a third time.
  3. Fix brain.conf (move return 301 into location /) → reload → certbot renew --cert-name brain.joaoluisbrazao.com.
  4. Add default_server catch-all returning 444 to nginx (closes the brain-on-raw-IP hole; orphan DNS records stop mattering).
  5. chmod 750 the two 777 root scripts; move LiteLLM master key into .env.

Phase 1 — This week

  1. Off-site backups: rclone config from template + cron + first-sync verification (C2).
  2. Rewrite restore-test with staleness check + Telegram alert (C3).
  3. Re-issue chat.joaoluisbrazao.cloud cert with container paths (H2 — hard deadline mid-July).
  4. certbot delete the 5 orphan lineages; disable host certbot timer+cron; systemctl disable nginx.service.
  5. Gateway async-await fix → revives all 5 JBSB engines (H5); fix podcast-orchestrator path (H6); fix graph-rebuild escaping (H7); fix ghrepos scanner env (H8).
  6. Rebind :3001/:8140/:8141 to 127.0.0.1 + route via nginx (H10). Optionally add Hostinger panel firewall rules as second layer.
  7. sshd: PasswordAuthentication no in 00-hardening.conf (M2).
  8. Cron hygiene: delete 3 duplicate/dead root-crontab lines, fix the 6 small broken crons (M11/M12), move Apify restore file out of /tmp.
  9. Decide + act on M17 (BYPASS_TOKEN) — it contradicts yesterday's login-fix decision.

Phase 2 — Maintenance window (schedule ~30 min downtime)

  1. Pre-reboot: host nginx disabled (done in P1), confirm rabbithall path rule, snapshot via Hostinger panel.
  2. Reboot → kernel 6.8.0-117; verify logrotate/man-db clear; verify 41 containers healthy.
  3. Recreate headscale (inherits log cap); raise litellm memory to 3–4 GB; retag gamine-db image; pin the worst :latest tags; add the missing healthchecks/CPU limits (compose edits batched here).

Phase 3 — Cleanup & decisions (João's call where marked)

  1. Delete the §5a file list (~12.3 GB) — decision on the March full-server tarball and /opt/docs password inventory.
  2. Docker: rm 6 empty networks, 3 stale volumes, builder prune (~1 GB). Keep audit-trivy-cache.
  3. Postiz decision (#735): resume (fix Prisma scoping) or remove stack+DBs+images.
  4. Skills: delete dead-legacy list, archive never-wired list, consolidate the 3 six-eyes scorers, fix/remove the tcn.deep-research ghost button, fix the silently-skipped taxonomy-extractor path.
  5. Screens: 1 dashboard card for the DARH island (#878), editorial-queue + publication + tcn-requests into nav, contacts-vs-CRM decision, re-skin 3 dark screens, purge ~40 .bak templates, retire the legacy discovery cluster if redundant.
  6. DNS: delete gestso-sandbox/ftp/brain-CNAME; decide the 7 unused records; add the 3 missing ones (#714/#565/#906) when ready.
  7. DB hygiene: SpendLogs retention (M8), chunks 1536-dim column drop (M9, ~1 GB), password rotation + scoped roles (M10), jb-brain dev/prod fix (M13).

Documentation corrections (CLAUDE.md §12)

  • ~~"SSL .com + .cloud expire 2026-06-28 URGENTE"~~ → renewed to 2026-08-27. Replace with: "brain cert expires 2026-07-02 — renewal broken (#887, now urgent); chat.cloud renewal conf broken (deadline mid-July)".
  • "Brain Content Curation Pipeline LIVE" → currently DOWN (640 perms) since ~May 31.
  • "JBSB Intelligence System LIVE" → engines have never run successfully (gateway async bug).
  • "#878 add UIs to dashboard nav" → confirmed still open.
  • "#59 volume backups empty" → confirmed real; root cause = wrong volume names in backup.sh.
  • Add: "backup.sh + restore-test + offsite — repaired 2026-06-XX" once Phase 0/1 lands.

Appendix A — Container inventory (45)

Container Image Status Health Mem limit Stack
rabbithall rabbithall:latest Up healthy 256M /opt/rabbithall-compose ✅
jb-skills-gateway jb-skills-catalog-* (1.29GB) Up healthy 1.25G /opt/prod/jb-skills-catalog
vaultwarden vaultwarden/server:1.35.4 Up healthy 256M password-manager
dr_orchestrator deeper-research-orchestrator:0.1.0 Up healthy 512M /opt/deeper_research
dr_worker_1/2/3 deeper-research-worker:0.1.0 Up — (no HC) 512M, no CPU cap /opt/deeper_research
dr_libretranslate libretranslate:latest ⚠️ Up healthy 2G, no CPU cap /opt/deeper_research
dr_searxng searxng:latest ⚠️ Up healthy 256M /opt/deeper_research
er-backend expense-report-backend:0.1.0 Up healthy 768M /opt/expense-report
er-celery / er-celery-beat expense-report-backend:0.1.0 Up HC disabled ⚠️ 1G/256M /opt/expense-report
er-frontend expense-report-frontend:0.1.0 Up — 256M /opt/expense-report
er-minio minio 2025-04-08 Up healthy 512M /opt/expense-report
infra-postgres infra-postgres-age:16-age1.5.0 Up healthy 2G infrastructure (vol 5.44GB)
infra-nginx nginx:1.29-alpine Up healthy 128M /opt/infrastructure/nginx
infra-redis redis:7 (rolling) Up healthy 128M infrastructure
infra-certbot certbot (latest ⚠️) Up — 256M infrastructure
infra-telegram-webhook infra-telegram-webhook:1.0 Up — 64M infrastructure
litellm litellm:main-stable (rolling) Up healthy 2G ⚠️ OOM history /opt/infrastructure/litellm
jb-brain-api jb-brain-api:0.1.0 Up healthy 512M /opt/dev/jb-brain ⚠️ (naming)
brain-mail-intake jb-skills-catalog-* Up healthy 128M jb-skills-catalog
darah-api darah-api:latest ⚠️ Up healthy 512M /opt/darah-compose
ghrepos-api ghrepos-api-* Up healthy 256M /opt/ghrepos-api
forgejo forgejo:10 (rolling) Up healthy 512M /opt/forgejo
admin-dashboard nginx:1.29-alpine Up healthy 64M /opt/admin-dashboard
dashboard-refresh dashboard-refresh-* Up — 64M /opt/dashboard-refresh (mounts docker.sock ro ⚠️)
gamine-web gamine-web (7.61GB ⚠️) Up — 256M /opt/gamine
gamine-db UNTAGGED postgres:16-alpine ⚠️ Up healthy 512M /opt/gamine
gamine-minio minio 2025-09-07 Up healthy 512M /opt/gamine (vol 4.36GB)
gamine-brain-api gamine-brain-api:0.1.0 Up healthy 512M /opt/dev/gamine-brain — ⚠️ port 8140 public
gamine-brain-web gamine-brain-web:0.1.0 Up healthy 128M ⚠️ port 8141 public
chatwoot-web chatwoot:v4.13.0 Up healthy 1G /opt/chatwoot
chatwoot-sidekiq chatwoot:v4.13.0 Up — 512M /opt/chatwoot
jb2b-api jb2b-* Up healthy 512M /opt/jb2b
vikunja vikunja:0.24.6 Up healthy 256M /opt/vikunja
uptime-kuma uptime-kuma:1.23.13 Up healthy 256M /opt/uptime-kuma — ⚠️ port 3001 public
headscale headscale:0.23 Up — 256M docker-run style; ⚠️ 57MB uncapped log
clamav clamav:1.4 Up healthy 2G security
falco falco:0.39.2 Up healthy 512M security
falcosidekick falcosidekick:2.34.0 Up — 128M security
postiz / postiz-elasticsearch / postiz-temporal / postiz-temporal-ui v2.21.7 / ES 7.17.27 / temporal 1.28.1 Exited since 2026-05-30 — — /opt/postiz — paused (Prisma↔AGE), decision pending #735

Totals: 45 containers (41 up), 22 compose stacks (21 running), 42 images (35.29 GB), 26 volumes (14.78 GB), 16 networks (6 empty). Postgres: infra-postgres 14 user DBs ≈5.1 GB; gamine-db 1 DB 20 MB.

Appendix B — Cron health (≈45 active jobs)

Working (18 verified): brain-sync (10 min), brain-analyze, brain-digest, brain-cluster, brain-contradict, content-suggest ×2, intelligence-synthesizer, tcn-discovery, tcn-six-eyes-scorer, topology_sync, taxonomy-realign (30 min, flock), jbsb.memory-decay, six-eyes-triage, vps-health-check (hourly), docker-image-prune (daily), jb-audit weekly+monthly (cron.d), architect-mentor (cron.d), certbot.timer (host — but redundant, see M5).

Broken (17): backup.sh (CRITICAL), backup-restore-test (CRITICAL), brain.content-curator ×2 (perms), jbsb engines ×5 (async bug), podcast-orchestrator (path), graph-rebuild (escaping), ghrepos scanner (env), geopolitics refresh (dropped DB), content-nuggets + brain-score-batch (renamed DB), deepseek-balance (source/dash), gamine_angle_cron (stale IP), publication-brief (enum cast), brain-lint (timeout ×4).

Duplicated (3): weekly audit, monthly audit (dead twin), architect.reporter — root crontab vs /etc/cron.d. Keep cron.d.

Suspended (5): Apify lines, correctly commented; restore copy must leave /tmp before reboot.

Audit artifacts: full agent output at the session workflow file; this report is the canonical summary. Generated by Claude Code CTO-audit (9 domain auditors + adversarial verification), 2026-06-10.