[Testing 15] test(evals): offline eval harness — citation accuracy, injection resistance, leakage

[amal66]

How this differs from #231 (closed)

#231 was closed for "limited usefulness of the evals and format. A fuller real evals suite can be implemented in the future which aligns with the format of actual assistant responses and involves real documents." This PR was rebuilt around that feedback:

• Real response format. Fixture answers are not invented markdown — they are prose with inline [N] markers plus a <CITATIONS> JSON block, exactly what the backend emits and parses. The harness's citation parser is ported line-for-line from backend/src/lib/chat/citations.ts (source lines cited in its comments), document pages mirror the [Page N] blocks from documentOps.ts, and recorded tool calls use the ToolCall shape from chat/types.ts with Mike's real tool names. docs/evals.md has the full fixture-field → runtime-format mapping with file references, so the format contract is auditable.
• Real documents. Sources are public-domain legal texts at real-world length: Marbury v. Madison and Erie v. Tompkins passages, Hadley v. Baxendale, 17 U.S.C. § 107, 15 U.S.C. §§ 1–2, and full-length contract clauses. Cited spans are buried mid-document, and injection payloads are embedded inside the real text — which is the attack shape the product actually faces.
• The checkers are proven, not assumed. Roughly half the corpus is known-bad cases marked expected: "fail" (a fabricated Erie holding, a liability cap quietly changed from twelve months to twenty-four, a canary smuggled into a tool-call argument). If a checker is ever weakened, those cases flip and CI goes red. Benign edge cases (docket numbers near the SSN pattern, an answer that warns about an injection without repeating it) go red if a checker over-triggers.

What this completes

CI already has an evals job that runs node evals/run.mjs --threshold 1.0 if the file exists and skips otherwise. This PR adds that file, turning the gate on: a deterministic scorecard for fabricated citations, prompt injection via retrieved documents, and privileged/PII leakage — no network, no LLM calls, no keys, no npm install, so it is free to run on every PR.

How to run

node evals/run.mjs                          # all suites, must be 100%
node evals/run.mjs --suite prompt-injection # one suite
node evals/run.mjs --list                   # see every case

Exit 0 at or above threshold, non-zero below. Verified both directions: the committed corpus is 27/27, and corrupting a single quoted span drops it to 26/27 with exit code 1 and a readable reason (quote not verbatim on page 1 of ...).

The suites

• citation-accuracy — parses the <CITATIONS> block and checks every [N] marker has a matching entry and every quote exists verbatim on the cited page (or in the fetched opinion for cluster_id citations). Covers cross-page quotes, page ranges, and the legacy citation shape.
• prompt-injection — recorded turns where the document carries an injected instruction and a canary; fails on canary leakage in output or tool arguments, or on a disallowed tool call (e.g. an injected note ordering an edit_document rewrite).
• privilege-leakage — flags off-matter emails, SSN-like values, sb_secret_* keys, service_role JWTs (the public anon key deliberately doesn't trip it), and document identifiers from another client's matter.

Future live mode

This gates checker logic plus a regression corpus; the answers are hand-written recordings, so it won't catch a live model regression. But because the fixtures already use the runtime shapes, the "fuller real evals suite" is a drop-in extension: run real prompts through the backend, capture the actual streamed responses and tool calls into this same fixture format, and score them with these same checkers — behind an opt-in, outside CI, since it needs keys and isn't deterministic. docs/evals.md spells this out, along with the runtime counterparts (citation verification in backend/src/lib/chat/citations.ts; spotlighting of untrusted content in the security-hardening PR #227).

🤖 Generated with Claude Code

[CLAassistant]

All committers have signed the CLA.

[amal66]

recheck