feat: durable job queues for conversion, extraction and embedding (ADR)

[amal66]

Design (ADR)

Summarized from the fork's docs/adr/0003-default-synchronous-env-gated-async.md, docs/adr/0004-database-enforced-idempotency.md and docs/async-jobs.md.

Context. Two workloads are expensive and can outlive a single request: DOCX→PDF conversion and tabular cell extraction. Running them inline is simple and needs no infrastructure, but the work dies with the request (a client disconnect or server restart loses it) and can't retry. Making them async needs Redis and workers — infrastructure a self-hoster kicking the tires should not be forced to run.

Decision. Every async-capable path is synchronous by default and opt-in to async via an ASYNC_* env flag, each defaulting to "false":

• Off → the work runs inline on the request thread (the historical behavior); no Redis required. The server only starts workers when anyWorkerEnabled() is true, so the default process has no Redis dependency at all.
• On → the work is enqueued to a BullMQ queue and an in-process worker drains it, gaining durability across disconnect/restart and retry-with-backoff (attempts: 3, exponential). Bulkhead concurrency caps per worker (conversion 2, extraction 3) bound resource use.
• The same core function serves both modes — extractDocumentColumns is called by the sync route and the async worker, differing only in missing-cell policy — so async is a deployment choice, not a rewrite.
• Correctness is pushed into the queue's identity model: each queue derives its jobId deterministically from the work identity (versionId for conversion, (reviewId, documentId) for extraction), so a double submit (client retry, reconnect) collapses into the in-flight job instead of running twice. Durable state lives in DB rows (documents.status, tabular_cells), not queue history; extraction jobs re-read cell state and only process columns not already done, so retries never redo finished work.
• The async /generate request becomes a reconnectable view: it subscribes to the review's Redis progress channel before enqueuing, forwards the same cell_update SSE frames the sync path emits, and a 3-second DB-poll backstop reconciles missed pub/sub frames so a dropped message can never leave the stream hung. GET /:reviewId/generate/stream lets a dropped client resume without re-triggering work.

Consequences. Trivial default onboarding (fresh clone runs everything inline); durability and retries where they matter; two code paths kept honest by sharing one extraction core; workers run in-process by default (split into a dedicated process by calling startWorkers() from a separate entrypoint when scaling apart).

Alternatives considered. Always-async (rejected: forces Redis on every self-hoster); app-level dedupe/locking for double submits (rejected: racy across replicas — the queue's jobId uniqueness and the DB are the only single points of serialization); a cron/poller instead of a queue (no retry semantics, no backpressure, higher latency).

Summary

Fewer stuck documents and fewer lost review runs. Today, a large DOCX upload blocks its HTTP request on LibreOffice, and a tabular review generation dies if the browser tab closes or the server restarts mid-run — the grid is left with spinners that never resolve. With the flags on, uploads return immediately and convert in the background with retries, and review extraction survives disconnects and restarts, retries transient LLM/storage failures with backoff, and lets a client reconnect to a running generation and catch up. Failures become explicit terminal states (document error, cell error) instead of hangs.

Changes

• backend/src/lib/queue/ — shared lazy Redis connection (BullMQ-safe maxRetriesPerRequest: null), conversionQueue, extractionQueue (deterministic jobIds, retry/backoff, bounded history), and runProgress (Redis pub/sub bridge for live cell updates).
• backend/src/workers/ — conversionWorker, extractionWorker, a declarative WORKER_REGISTRY, and startWorkers()/stopWorkers() lifecycle.
• backend/src/lib/tabular/ — the extraction core factored out of routes/tabular.ts so the sync route and the async worker share one loop: tabular.extract.ts (LLM cell extraction + PDF/DOCX/Office text extraction), tabular.extractDoc.ts (extractDocumentColumns), tabular.generate.ts (prepareTabularGenerate guard), tabular.generateStream.ts (async enqueue + reconnectable SSE tail), tabular.shared.ts, tabular.prompt.ts. routes/tabular.ts now imports these instead of its inline copies (moved, not changed — bodies are byte-identical apart from import/type-annotation mechanics).
• backend/src/routes/tabular.ts — POST /:reviewId/generate gains the async path behind ASYNC_TABULAR_EXTRACTION; new GET /:reviewId/generate/stream resume endpoint. Inline path remains the default.
• backend/src/routes/documents.ts — upload defers Office→PDF conversion to the queue behind ASYNC_DOCUMENT_CONVERSION (doc stays processing until the worker flips it to ready).
• backend/src/index.ts — start workers only when a flag is on; graceful SIGTERM/SIGINT shutdown (drain server, stop workers, close queues + Redis, 15s force-exit guard).
• backend/src/lib/sseHeartbeat.ts, backend/src/lib/pdfjs.ts — small helpers the above use (SSE keepalive comments; typed facade over pdfjs-dist).
• backend/.env.example, backend/tsconfig.json (exclude tests from tsc, as the fork does), backend/package.json + lockfile.
• Tests: lib/queue/__tests__/{conversionQueue,extractionQueue}.test.ts, workers/__tests__/{conversionWorker,extractionWorker}.test.ts, lib/tabular/__tests__/{tabular.extractDoc,tabular.generateStream}.test.ts.

Why

Leads with the cost posture: both flags default "false", and with them off the server never dials Redis — behavior and infrastructure requirements are exactly what they are today. A cost-sensitive firm on a £20 VPS changes nothing and pays nothing extra. Turning a queue on needs only a Redis container on the same box (REDIS_URL, defaults to redis://localhost:6379).

New runtime deps (both shipped by the fork for this feature): bullmq ^5.34.0 (the queue) and ioredis 5.10.1 (its Redis client; the lockfile pins bullmq 5.79.2, matching the fork's lockfile, so a single deduped ioredis serves both). No other dependency, schema, or API change.

Note on the title: the fork's third queue (document embedding) is not in this PR — its queue/worker import the RAG ingest module, so that wiring rides in the follow-up RAG PR where the dependency is real.

Testing

• npm install && npm run build (tsc) green on the branch as committed.
• With the vitest harness merged locally (not part of this branch): npx vitest run → 7 test files, 39 tests, all passing (the 6 ported suites above plus the harness's existing downloadTokens suite).
• Flags-off path: the queue modules are only imported by the gated call sites; nothing constructs a Redis connection unless a flag is "true" (connection is lazy and only reached via enqueue*/startWorkers).

Provenance

All added lines are mechanical ports of amal66/mike@origin/main (b3166dd) — path moves (apps/api/src/* → backend/src/*, modules/tabular/* → lib/tabular/*), import rewrites, and inlined types; exceptions:

• The fork's zod-validated lib/env.ts module is not ported; env reads are inlined as process.env.X with the fork's defaults preserved (ASYNC_* default "false", REDIS_URL default redis://localhost:6379).
• The fork's observability infrastructure is not ported: pino logger calls are translated to console.*, and OpenTelemetry trace propagation (withTraceContext, the otel?: OtelCarrier payload field, runWithRequestContext/withExtractedContext wrappers) is removed from queue payloads and worker callbacks. The Log type becomes Pick<Console, "error">.
• tabular.prompt.ts ports only formatPromptSuffix (identical to the inline copy this PR deletes from routes/tabular.ts); tabular.generate.ts ports only prepareTabularGenerate; tabular.extract.ts omits the fork's extractTabularAnnotations and the redline-summary enrichment of extractDocxMarkdown — those depend on fork-only chat/tracked-changes code, so the ported body matches upstream's existing inline function exactly.
• routes/tabular.ts deletions are the inline copies of the moved helpers (verified byte-identical modulo import/type annotations); the new /generate body is the fork's route body with req.log → console.
• Test files: the fork's vi.mock("../../env", …) shims are dropped (no env module here) and mock import paths follow the path moves.
• bullmq/ioredis version pins match the fork's package.json + lockfile resolution.

Credits & prior art

• @nwhitehouse (nwhitehouse/mike) — independently parallels this work: their fork built a durable job worker pool (plus an agent event audit log and loop controller) for long-running work. Different implementation (this PR is BullMQ/Redis with env-gated sync fallback), same conclusion that request-lifetime execution loses work.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CEguyEgXa9JjCciXCcVemC