Shell Heuristics · application docs

publisher · enricher-v2 · corpus

Postcrime/Precrime Enrichment — Proposed Key Tasks (2026-06-08)

Synthesized from: live progress board (/api/progress, 54 rows), ~/.claude/TODO.md (topics 2/3/6), repo memory pack, and the in-flight working tree on stats/rework.

Current state (observed, not assumed)

  • Branch stats/rework: 21 unpushed commits + 12 files dirty mid-refactor.
  • Enrichment pipeline is LIVE + self-healing (P0 broker-volume-loss fixed; corpus already ~95–100% enriched). Most enrich.* tasks sit running 0/0 = saturated, by design.
  • Ingestion is the live risk: postcrime-crawler-doj-sdny and postcrime-crawler-sec-pr flapped to error on the board (transient on re-poll). New data only arrives via crawlers.
  • gpumon:federation-scanner stuck running 0/8. All precrime/edgar-scan-shard0-8 idle; precrime schema empty.
  • In-flight uncommitted change = purpose-pool routing refactor: caller names workload purpose (pool-enrich-facts/entities/titles/summarize), gpumon owns model→pool mapping; retired model-family aliases (pool-postcrime-qwen-14b, pool-qwen-large, pool-gemma). Plus extract-case-facts now covers press_release+pdf kinds (~6.3k docs that had no facts).

FIX (do first — protect in-flight work, restore ingestion)

  1. Finish + commit the purpose-pool routing refactor (12 dirty files). Half-done = loss risk. Pre-deploy gate: confirm pool-enrich-{facts,entities,titles,summarize} actually resolve on gpumon-ingress (else tasks 5xx into the spillover sink). Then deploy + grep a marker in the running container on node-eighteen (image is local-cache-only, no registry).
  2. Crawler error recovery — root-cause the doj-sdny / sec-pr flaps; add restart/backoff + board error→running reconcile. Enrichment is saturated; ingestion is the only growth lever.
  3. federation-scanner 0/8 stall — investigate stuck progress; emit idle/error not silent.
  4. Coverage-display bugtask_state.done undercounts (populated=0 = NULL column, not lost data). Backfill done-rows from existing columns so /stats + board stop under-reporting.

IMPROVE (throughput + hygiene)

  1. P0 pool-fallback-on-retry (processor/src/llm.ts:202-285, TODO topic 2): carry fallbackPools, swap x-gpumon-pool on 5xx/408/quality-flag. Largest single lift (30→70% success).
  2. Commit + push the backlog — 21 unpushed commits + topic-3 scheduler self-heal/ops scripts if still uncommitted. Long-lived unpushed branch on the only push host is a real risk.
  3. Config-version drift reconciledocker/stack.yml vs live (pools2_v52) BEFORE any gpumon stack deploy. Deploy landmine; prereq for #1 and #5 redeploys.

BUILD (forward roadmap)

  1. Phase B remainder (TODO topic 3 item 5): B2 CIK-classify, B4 graph/distance/scheme scoring, B5 CourtListener edges, B6 stats pages, B7 precrime bundling.
  2. Activate precrime — all edgar-scan-shard* idle, precrime schema empty. Wire the forward- looking EDGAR fingerprinting scan to actually produce rows (the precrime half of the umbrella).
  3. Unblock the rename-consolidation (topic 6) — gates the enrichment umbrella; do after #1–#7 stabilize. Also clears the decommission tail (n22 snapshot-server, nv1 corpus-pull timer feed nothing).

Suggested batching

1 → 7 → 5 (protect refactor, reconcile config, ship throughput) is the critical chain. 2, 3, 4 are independent fixes droppable between. 6 anytime. 8/9/10 are the build roadmap.