OCR pipeline
How a scanned/image-only PDF becomes searchable text. OCR runs off the enrich-runner through the federation gpumon-ocr-api gateway (the in-process cascade was retired 2026-06-09 — it OOM'd the runner on multi-page PDFs). The runner only submits and later reaps; the heavy work is on the gpumon OCR fleet.
Two-stage flow
Stage 1 — submit (enrich-runner). The recover-ocr task loads the PDF bytes
(local /data NFS copy, falling back to the SeaweedFS filer), POSTs them to
gpumon-ocr-api, and records ocr_job_id + ocr_status='queued' on the
document row. It does no rasterization or OCR itself, so it returns in ~1s and
never spikes memory.
Stage 2 — reap (enrich-runner). The in-process ocr-reaper polls rows in
ocr_status IN ('queued','running'), asks the gateway for each job's state, and
on done fetches the result text → writes body + ocr_text, sets
ocr_method='gpumon-ocr-api' + ocr_status='done', and clears the
[OCR_UNRECOVERABLE] sentinel. failed / empty / gone-after-retries → failed.
In between, the gpumon OCR fleet does the actual work: validate → IPFS → queue → rasterize → per-page vision (or tesseract) OCR → LLM cleanup → assemble.
Components
| Component | Host / stack | Role |
|---|---|---|
enrich-scheduler |
postcrime-enrich · node-eighteen | Emits enrich.recover-ocr jobs (new-doc / backfill) to RabbitMQ. |
enrich-runner → recover-ocr |
node-eighteen | Loads PDF bytes (/data → SeaweedFS filer), submitPdf() to the gateway, records ocr_job_id/ocr_status='queued'. Skips docs already queued/running (no double-submit). |
gpumon-ocr-api :2295 |
gpumon | Producer. Basic-auth → validate (magic-byte + pdfinfo) → upload PDF to IPFS (kubo, unpinned) → POST writer /api/ocr/jobs/start (durable row) → publish envelope v1 to federation.work routing-key ocr.jobs → 202 {id}. Never sees PDF bytes after the IPFS upload; reads proxy to the writer. |
| IPFS / kubo | gpumon | PDF blob transport — the api uploads, the worker fetches by ipfs_cid. |
RabbitMQ ocr.jobs |
gpumon-rabbitmq · node-eleven | The OCR work queue (federation.work exchange, vhost federation). |
gpumon-ocr-worker ×2 |
node-eleven | Consumer. Fetch PDF from IPFS → runPipeline: pdftoppm rasterize → page chunks (≤200 pages) → per-page OCR → chunked cleanup → assemble txt+md → POST writer /api/ocr/jobs/finish. AMQP heartbeat 600s; per-page retries ×20; AMQP retries ×5 then DLQ. |
pool-ocr |
gpumon ingress | Vision OCR (NIM nemotron-vl + spark qwen2.5-vl). One page image → transcribed text. |
pool-doc-cleanup |
gpumon ingress | Cleanup LLM (NIM). Conservative: fixes OCR spacing / hyphenation / artifacts, 6 000 chars/chunk — no rewriting. |
doctor swarm :7860 |
gpumon · node-eleven | Tesseract backend (CPU) — used when the job is submitted backend=tesseract. |
gpumon-writer :2289 |
gpumon | Durable job state (start / progress / finish / fail) + result store; serves /jobs, /jobs/:id, /jobs/:id/result.{txt,md,json}. |
gpumon-ocr-cache |
gpumon | Result cache (content-addressed) so identical PDFs skip re-OCR. |
ocr-reaper (in enrich-runner) |
node-eighteen | Polls pending rows → getJob; on done getResultText → writes documents.{body,ocr_text,ocr_method,ocr_status} + clears the sentinel. Idempotent across replicas. |
postcrime.documents |
.181 ParadeDB · pve-seven |
Target table. ocr_job_id / ocr_status / ocr_attempts / ocr_submitted_at track the async job; ocr_text / ocr_method / ocr_confidence hold the result. |
Job status lifecycle
queued (submitted) → running (worker rasterizing/OCR'ing) → done (reaper
wrote text) — or → failed (gateway failed, empty result, or the job id went
unknown after 5 reaper retries). The partial index idx_documents_ocr_pending
keeps the reaper's scan cheap.
Separation of concerns (current vs target)
Today the gateway's per-chunk LLM cleanup (pool-doc-cleanup) is an
enrichment step folded into OCR. The target architecture keeps OCR to
rasterize + tesseract + vision + embeddings only and emits a per-page,
per-engine JSON envelope (text_layer + tesseract + vl_ocr per page),
with cleanup / bluebook / summaries / scheme as separate downstream services. See
the design page ocr architecture under Design / v2.