Repository navigation
Never certify an absence over an incomplete search (false-absences theme + #17) - #21
Merged
Merged
Conversation
CourtListener began answering non-browser clients with an AWS WAF JS-challenge: HTTP 202, an empty body, and `x-amzn-waf-action: challenge`. The challenge names no vendor in its markup, so every _SIGNATURES pattern missed it -- and fetch() discarded response headers entirely, so the one place the mechanism WAS named never reached detect_blocker. The empty body therefore flowed downstream as content, and `extract` printed "the document was read, the terms are not in it" over a fetch that read nothing. That is a false negative manufactured by the tool, which is the single failure this package exists to prevent. Reported 2026-08-28 with a correct curl root-cause and a written spec, and again on 2026-09-17 by a worker who re-derived the whole diagnosis because nothing marked the first report as known. The second reporter noted the only tell was a `raw ~1 tok` line they happened to have a successful `raw ~54,024 tok` fetch above in scrollback to compare against. detect_blocker now reads headers and classifies the pair (challenge-shaped response, no visible prose) as AWS_WAF, which is browser-escalatable and names the retry in its detail. Detection is never on size alone: a 202 carrying real prose is served, and a legitimately tiny page stays a Hit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dollar_sum counted only rows whose `Award Amount` was numeric, then
reported "no priced rows" as a VerifiedAbsence at exit 11 -- which a shell
caller reads as publishable. `n == 0` conflated two different facts: the
recipient has no awards, and the recipient has awards we failed to parse.
Feeding two real awards whose amounts arrived as strings ("5,000,000" -- a
shape this module already defends against in _parse_counts) certified a
$7.5M vendor as having no federal awards, with claim() == "Not found".
The Coverage guard could not catch it. That guard refuses coverage a
source DECLARES dirty; here the source built a clean Coverage over a
failure it never noticed. dollar_sum is also the one entry point that
re-implements the run_source shell by hand, which is why it carried a bug
every shell-routed source was structurally protected from.
Unparseable rows and an unread `hasNext` page are now AccessBlocker. A
genuine absence still certifies, and now carries probes plus the subaward
caveat instead of the bare, unauditable "Not found".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cache stored only result rows, so replay_cached rebuilt a VerifiedAbsence without its probes, not_searched or caveats. claim() degraded from a scoped sentence to the bare string "Not found" -- same question minutes apart, same exit 11, materially weaker claim. That is the opposite of the README's own worked example, where the published sentence survives a lawyer precisely BECAUSE the tool printed the subaward caveat: "without it the sentence is an overclaim a lawyer could take apart." The tool produced the parenthetical on a cache miss and dropped it on a cache hit, within a 24h TTL (7 days for crossref). Severity was bounded -- is_absolute requires bool(probes), so a scopeless replay could never read as an absolute negative -- but the caveats are the part a reporter copies into the piece. Scope now persists in a `__rows__`/`__meta__` sidecar rather than a schema change, so rows written by an earlier version still read as bare lists and simply replay the weaker claim they were stored with. Staleness disclosure is unchanged: a replayed absence still says when it was established. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-source absence tests are hand-written spot checks, and they are only as complete as the list someone thought to write. That is how usaspending.dollar_sum shipped a false absence with 209 tests green: nobody spot-checked the one function that re-implements the shell by hand, so nothing enforced the invariant there at all. This walks EVERY registered entry point against transport failure, rate limiting, and a malformed success payload, asserting that none yields a VerifiedAbsence. The point is the next source: a contributor who adds one and forgets its absence test still gets the sweep, and one who hand-rolls their own Coverage -- the thing that caused two of the three live bugs -- fails here. Two entry points are xfail rather than hidden. oscn.search and docs.search scrape HTML, and a scraper counting rows genuinely cannot tell "the page served no results" from "this is not the page I know." Fixing that needs a positive shape check of the kind oscn already has for its real empty page. Tracked, not fixed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ALLOWED_HOSTS was a hardcoded set of 18 with no override, so reaching any new government registry required a code commit and a release. Agents filed 46 reports naming hosts they could not reach -- loc.gov (12 mentions), fec.gov (9), justice.gov (5), apps.occ.gov, scgov.net, county clerks -- while the error message told them to "add it to ALLOWED_HOSTS deliberately" and gave no route to do so. Two reports asked for the list to be queryable. Meanwhile the two hosts that did get added appear nowhere in FEEDBACK.md, so the only code changes in a month were not the ones being asked for. A hardcoded set is the right default and the wrong only option. Operators now extend it in ~/.evidence-search/allowed_hosts.json (or $EVIDENCE_ALLOWED_HOSTS), and `evidence-search hosts` prints the current boundary. The ethical boundary is unchanged and tested. A reason is REQUIRED per host -- an entry without one is refused, so the file cannot decay into an undocumented pile of hostnames, and CONTRIBUTING's "adding a host is a deliberate act" still holds. The match is still the dot: allowing sos.example.gov never allows evil-sos.example.gov. A malformed config fails closed to the shipped baseline, which config can extend but never reduce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CONTRIBUTING claims field reports drive fixes better than synthetic tests. That was true, and the seed-era triage proves it -- but the loop ran one-directional for a month: reports kept arriving and almost none became code or tests. Every [fixed]/[wontfix]/[tracked] marker sat in the seed era while entries accumulated untriaged, and the file's own header went on claiming everything below it was marked. The cost is measurable: the CourtListener soft block was reported with a correct root cause and a written spec, went unmarked, and was re-derived from scratch weeks later by a worker who paid the whole diagnostic cost again. Triages the two recurring clusters (web rate limiting, browser allow-list gaps), corrects the stale header, and appends the triage pass. Original entry text is left intact, per the file's own rule. Adds tests/test_feedback_hygiene.py so this is mechanical rather than remembered: one test tripwires when the untriaged backlog passes 40, the other fails if the file asserts a triage it has not done. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
) `usaspending "70CMSW26D00000016" --keywords --count` exited 11 with a VerifiedAbsence. The award exists, as an IDV; `--all-types` finds it. The CLI counts contracts only by default, and counts() never said which award groups it had skipped, so a zero over contracts read as a zero over every federal award. Same invariant as the rest of this branch: a negative must state its own scope. - counts() lists every award group it did not search in not_searched. - A PIID-shaped query routes by its 9th character, the instrument type: A/B/D (BPA, BOA, indefinite-delivery) are IDVs, so a count that left IDVs out now includes them, with a caveat saying so. Widening a count can find more but can never manufacture an absence. - Q/R mark a solicitation, which USAspending cannot hold. counts(), search() and dollar_sum() return a WRONG_ID_TYPE AccessBlocker pointing at SAM.gov before spending a ledger slot, never an absence. Regression tests in tests/test_every_source_absence.py, including the live 70CMSW26D00000016 case. Fixes #17 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Florida Division of Corporations (Sunbiz) serves a Cloudflare interstitial to a plain fetch; Kentucky SOS business search is an ASP.NET postback form; Schuylkill County PA and Berks County PA publish real-property records behind JS-rendered search pages. All four are government public-records systems -- business registries and property records -- the category the allow-list exists for. Each entry carries its reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FEEDBACK.md is tracked in a public repository, and agents write their field reports mid-task -- naturally naming the task, the subject, the ticket and their working paths. That is the right raw material and the wrong thing to publish. - FEEDBACK.local.md is gitignored and is where raw entries now go. - FEEDBACK.md takes redacted entries only, and says so in its header: people, investigations, vendors, committees and places become stable placeholders; tasks are described by kind, not ticket name; no local paths, KB ids, ticket or draft names. Public identifiers that are the bug (PIIDs, endpoints, government domains) stay. Every existing entry has been redacted to that standard; the report text is otherwise unchanged. - The evidence-search-usage skill tells agents to log raw to the local file and redact before anything reaches FEEDBACK.md or an issue. - tests/test_feedback_hygiene.py fails on a local path in FEEDBACK.md and on FEEDBACK.local.md dropping out of .gitignore. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
markramm
marked this pull request as ready for review
September 29, 2026 14:59
markramm
added a commit
that referenced
this pull request
Sep 29, 2026
…d keyless, issue numbers; add www.gao.gov to the browser allow-list Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The false-absences theme: never certify an absence over a search that wasn't complete. It adds #17, which is the same invariant, and keeps raw friction logs out of the public repo.
False absences fixed (each with a regression test)
AWS_WAF, which is browser-escalatable. Detection keys on response headers plus no visible prose.usaspending.dollar_sum: award rows whose amounts wouldn't parse were certified as "no awards" (exit 11). That is now anAccessBlocker, and the zero-rows-but-hasNextcase is fixed too.probes,not_searchedandcaveats. Scope now persists in a sidecar.tests/test_every_source_absence.pyruns every source entry point against transport failure, rate limiting and a malformed payload.oscnanddocsarexfail(HTML scrapers; tracked).counts():not_searched.A/B/D(IDV instruments) widen the count to include IDVs, with a caveat.Q/Rmark a solicitation:counts(),search()anddollar_sum()returnWRONG_ID_TYPEpointing at SAM.gov before spending a ledger slot. Never an absence.70CMSW26D00000016.Browser allow-list
~/.evidence-search/allowed_hosts.json, and a reason is required per host. The shipped baseline can't be reduced, andevidence-search hostsshows the current list.Feedback hygiene
[tracked]markers on the two recurring clusters, a corrected stale header, and a 2026-09-19 summary.FEEDBACK.local.md. The trackedFEEDBACK.mdtakes redacted entries only, and every existing entry has been redacted to that standard..gitignore. They also tripwire an untriaged backlog over 40.Review notes
fix/false-absencesbranch were cherry-picked ontoorigin/main. That branch sat on unpushed local-main friction logs that name private research, and those are not published here.--type rlackstotal_matches) is included in generalized form.--idv <PIID>child awards;--agency; a reporting-lag note. A listing (search) of a D-shaped PIID with the default--group contractsstill lists contracts only. Itsnot_searchednames the other groups, but it doesn't reroute.Tests:
./.venv/bin/python -m pytest -qgives 262 passed, 2 xfailed.Fixes #17
🤖 Generated with Claude Code