[Security 6/9] Server-side citation verification against source text

[amal66]

[Security 6/9] Server-side citation verification against source text

Part of the split of #227 into single-topic PRs. Index: tracking comment on #227.

TL;DR

Every quoted citation the model produces is checked server-side against the actual document text — pure deterministic string matching, no extra LLM calls. Quotes that match exactly are verified; quotes that match under whitespace/case/punctuation tolerance are repaired (and the exact source text is swapped in); quotes that don't appear are unverified. A missing status is treated as untrusted by the UI.

Risk to user data

Severity: high — integrity, not confidentiality. The adversary here is the model itself. LLMs fabricate plausible-looking quotes and attributions. When the app presents a quote as coming from a document, that is an integrity claim, and in legal work a fabricated citation is a sanctionable harm. This is also the third defense-in-depth layer against prompt injection ([5/9]): even if an injection makes the model assert something, a quote it attributes to a document must actually be in that document to be shown as verified.

Flows affected

• Chat / project-chat citation finalization (streaming.ts).
• New verifyCitations.ts runs after the model's <CITATIONS> block is parsed.
• Air-gap preserved: source text is read from storage with emitEvents:false, memoized per doc_id, at most one read per document per turn. Case-law citations (verified upstream via CourtListener) pass through untouched.

Attack precedent

• Mata v. Avianca, Inc. (2023) — lawyers filed a brief citing ChatGPT-invented case law and were sanctioned. (overview) The canonical example of hallucinated citations causing real-world harm.

Possible fixes, and what we chose

Option	Verdict
Trust the model's quotes	The status quo we're fixing. Hallucinations ship to users.
Ask a second LLM "did the first hallucinate?"	Rejected — inherits the same failure mode (a model judging a model), costs tokens/latency, and is itself injectable.
Deterministic string matching against the extracted source text	Chosen. A quote either appears in the source or it doesn't — no model judgment, no LLM calls, not injectable.

flowchart TD
    Q["model's quoted citation"] --> M{exact substring<br/>in source text?}
    M -- yes --> V["verified"]
    M -- no --> T{match under whitespace/<br/>case/punctuation tolerance?}
    T -- yes --> R["repaired<br/>(swap in exact source excerpt)"]
    T -- no --> U["unverified<br/>(preserve model text, flag untrusted)"]

Loading

Design subtleties: a repaired quote has the exact source excerpt substituted in, so drifted text is never surfaced as the source's words. Cross-page quotes (joined by a [[PAGE_BREAK]] sentinel) are split and each segment verified independently. Unreadable-source sentinels are treated as "no source" so a quote falls back to unverified rather than false-matching the error string. Custom builders (tabular) that bypass verification carry no status, and the UI treats a missing status as untrusted.

What's in this PR

• backend/src/lib/chat/verifyCitations.ts — the tiered matcher (new).
• backend/src/lib/chat/types.ts — CitationVerificationStatus / QuoteVerification.
• backend/src/lib/chat/streaming.ts, tools/documentOps.ts — wire verification into finalization.
• Tests: verifyCitations.test.ts (16 tests: exact/repaired/unverified, cross-page, unreadable source).

Reading

Mata v. Avianca · OWASP Top 10 for LLM Apps — overreliance / hallucination

Known limitation (from a post-open adversarial re-review)

Tier-3 tolerant matching strips punctuation without inserting a separator, so digit tokens can collapse (1.2 → 12): a quote citing "Section 1.2" can tier-3 match a source that only contains "Section 12". Tiers 1–2 (exact and whitespace/case-normalized) are unaffected. The tight fix — inserting a separator when punctuation sits between digits — touches the shared normalizeWithMap used by find_in_document, so I'd rather land it as a small follow-up with its own tests than widen this diff. Flagging it now so the tier-3 tolerance is reviewed with eyes open.