Collector on Argus: Firecrawl retrieval, batched Claude classification - #74
Merged
Conversation
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ecrawl Candidate pages discovered from fetchStrategy 'browser' sources now try the self-hosted Firecrawl stack before falling through to the headless browser. Any Firecrawl ok:false (timeout, unavailable, http_error, empty) falls through to Playwright rather than aborting the candidate, keeping this reversible for one cycle. Playwright remains the fallback and is still used directly for each source's own index/feed retrieval, which needs real HTML or XML to extract links — Firecrawl only returns markdown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Batches candidates into groups of 20 for cost-efficient classification. Single claude -p invocation carries ~$0.25-0.44 fixed overhead, so batching reduces monthly cost from ~$375 to ~$13 at current volumes. Malformed entries are silently skipped; zero candidates trigger zero Claude invocations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ocess mock
vi.mock('node:child_process', …) in claude-cli.test.ts was not actually
intercepting execFile — Argus runs showed the real execFile firing (hitting
the CLAUDE_BIN=/bin/false guard: "Command failed: /bin/false -p test
--output-format json"). Before that guard existed this spawned the real
claude binary nine times in parallel from a unit test, making paid API calls
and crashing the host.
Replace the builtin mock with dependency injection: claude-cli.ts holds a
module-level `exec` (typed ExecLike) defaulting to promisify(execFile), with
an exported setExecForTesting() test seam. Public contract (runClaude,
ClaudeAuthError) is unchanged, so claude-classify.ts needs no changes.
Rewrote claude-cli.test.ts against the new seam — same 9 behavioural cases.
Also harden vitest.setup.ts to force CLAUDE_BIN=/bin/false for every test
run, so a future missed mock/seam fails instantly instead of silently
falling through to the real binary.
…exec test vitest.setup.ts now forces CLAUDE_BIN=/bin/false for every run; the flags test was still asserting the binary arg equalled 'claude' and failed against its own hardening. Assert against process.env.CLAUDE_BIN instead.
…ing score Classification.relevanceScore was doing two incompatible jobs: gating whether a Claude detection reaches the review queue (needs the real confidence) and feeding Development/SourceReview data an editor sees (must always read as needs-review, per the methodology page's framing of machine confidence as evidence, not verification). A verdict with confidence 0.99 was flowing uncapped into stored data. Split the two: relevanceScore is now capped at the same MACHINE_CONFIDENCE_CAP (0.65) the heuristic path already uses, so nothing persisted or editor-facing ever exceeds it. A new Classification.rawConfidence carries the uncapped verdict in memory only, used solely by collect.ts's review-creation gate so a genuinely confident Claude detection still reaches an editor instead of being silently capped out of the queue.
![policai-collector[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
AGENTS.md, README.md, docs/trust-model.md, docs/collector.md and the methodology page previously said analysis was deterministic keyword heuristics with no external AI provider. That became false once the collector started retrieving browser-strategy candidates through a self-hosted Firecrawl instance and classifying them with Claude (behind USE_CLAUDE_CLASSIFIER, enabled on Argus). Ships with the code change rather than after it because the site's whole purpose is being trustworthy about how Australian AI policy is assessed, and the old text was actively false, not just stale. Also documents that collection now runs on an Argus systemd timer in its own checkout, with GitHub Actions kept only as a manual fallback, and restates the confidence cap and policies.json guard against the new classification path.
… page
Review flagged that the methodology page hedged production reality behind
flag mechanics ("when the collector's classifier is enabled"), on the one
page whose job is candor about the site's own AI use, even though
USE_CLAUDE_CLASSIFIER is set in production and every visitor's data goes
through Claude. States it as fact, with the keyword heuristic named as the
fallback path, without naming the env var (operator detail, not reader fact).
…doc TODOs Prompt injection in scraped page content could otherwise reach Claude Code's auto-allowed read-only tools (Read/Glob/Grep) in headless mode and exfiltrate data through the persisted summary field. Add --tools "" (removes all built-in tools from the session — --max-turns does not exist on the installed CLI, verified via `claude --help` on Argus) plus an explicit --disallowedTools denylist as a second, independent layer, and pin the new argv in the regression test. Also converts two hosting-argus.md "Carried into Phase 2" bullets from open questions to resolved/past-tense now that Claude auth and the methodology doc rewrite are settled. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ion run Both collected 2026-08-07. Main's commit came from the then-scheduled collector, so its data stands; the branch's run was validation and its evidence lives in the PR description.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the design in #73 (
docs/collector-argus.md). Collection moves from GitHub Actions to the Argus server, with Firecrawl retrieval and Claude relevance judgment. The deterministic pipeline remains the spine and the default.What changed
firecrawl.ts— client for the self-hosted Firecrawl on Argus (port 3003 only; cold starts tolerated, not failed). Browser-strategy candidate pages retrieve through it, falling back to Playwright; index/RSS pages stay on Playwright, whose HTML the link extractor needs.claude-classify.ts/claude-cli.ts— batched classification through the Claude Code CLI (batch 20; a single invocation carries ~$0.25–0.44 fixed overhead, so per-candidate calls are ruled out by construction). Zod-validated; malformed entries dropped, never persisted.classify.ts/collect.ts— Claude path behindUSE_CLAUDE_CLASSIFIER(unset ⇒ deterministic path byte-identical, pinned by test). Raw model confidence drives the review gate; everything stored or editor-facing is capped atMACHINE_CONFIDENCE_CAP(0.65), so automation never reads as more certain than an editor.collect.yml— schedule removed;workflow_dispatchkept as fallback. File deletion follows after one clean Argus cycle.AGENTS.md,README.md,docs/trust-model.md,docs/collector.mdand the methodology page now describe what actually runs; 13 stale claims (including "no external AI provider") replaced with true statements. The methodology page names Claude, an Anthropic model, as present fact.Security
Final review caught that model isolation rested on CLI defaults. The invocation now enforces it:
--tools ''plus an explicit--disallowedToolslist, pinned by an argv regression test and verified against the real CLI on Argus (num_turns: 1). A prompt-injected scraped page cannot reach tools.Separately,
vitest.setup.tspinsCLAUDE_BIN=/bin/falseso no unit test can ever spawn the real binary — the failure that previously made real API calls from tests.Proven end to end
A real collection run on Argus (this branch, commit c185323 included here): 30/32 due sources reached, one live DTA item classified by Claude at capped 0.65,
data/policies.jsonbyte-identical (test + runtime hash guard),validate:dataclean, data commit pushed.Server side (already installed, dormant):
~/live/policai-collectorcheckout,policai-collect.service/.timer(19:30 UTC daily), failure alerting to a deduplicated GitHub issue with a distinct message for expired Claude auth. The timer stays disabled until this merges.Verification
442/442 tests, typecheck, lint,
next build(33 routes) — all on Argus. Review process: fresh implementer + reviewer per task, scoped re-reviews per fix, whole-branch final review; deferred minors triaged in the final pass (all judged safe to defer; the one Critical was fixed).After merge
policai-collect.timeron Argus.collect.yml.🤖 Generated with Claude Code