Design: move the collector to Argus (Firecrawl + Claude)

[l0cka]

Design only — no code changes. Phase 2 of the move to Argus, following #72.

Read: docs/collector-argus.md

Shape

Retire collect.yml. Run collection on Argus, retrieving through the Firecrawl stack already self-hosted on the box and using Claude for the relevance judgment that keyword heuristics do badly. collect.ts, dedup, watch-state, validate:data and the policies.json guard all stay, along with their ~3,300 lines of tests.

The two failures it targets

Measured, not assumed:

• Retrieval — 2 of 33 due sources unreached (health is degraded), plus 1 source marked manual because its publisher blocks automation.
• Judgment — 97 detections sitting in pending_review.

Four constraints on the model

These are the design, not caveats:

1. Claude receives text and returns JSON. No file, shell or network access — the pipeline does the writing.
2. Its output is schema-validated before persisting. Malformed output is a failed item, not a corrupt record.
3. Nothing it produces is verified. Model-touched records stay in the radar until an editor confirms them against the primary source.
4. data/policies.json is never touched. The existing guard stays and becomes the most important check in the pipeline.

Structural decision worth reviewing

The collector needs its own checkout. It cannot run in ~/live/policai, because policai-deploy.sh refuses to pull when local HEAD is not an ancestor of origin/main — a collector committing there would trip that guard every run, correctly. So it clones separately and pushes to GitHub; the serving checkout learns about new data by pulling, exactly as it does for a merged PR.

Operational notes captured

• Firecrawl must be called on port 3003, never 3002. 3002 is the raw API and is down whenever the stack idles; calling it directly produces connection-refused errors that look like source outages. This has already caused a false alarm on another project.
• Cold start is ~17.5s. The client uses a longer timeout for the first call of a run rather than treating it as failure.
• Argus can push to main directly: admin is a bypass actor on the "Protect main" ruleset.

Documentation is in scope, not a follow-up

AGENTS.md, README.md, docs/trust-model.md, docs/collector.md and the methodology page all currently state that analysis is deterministic with no external AI provider. That becomes false. Rewriting them ships with the code — on a site whose purpose is to be trustworthy about Australian AI policy, an inaccurate claim about its own use of AI is the most damaging defect available.

🤖 Generated with Claude Code