Skip to content

Never certify an absence over an incomplete search (false-absences theme + #17) - #21

Merged
markramm merged 9 commits into
mainfrom
fix/false-absences-17
Sep 29, 2026
Merged

markramm merged 9 commits into
mainfrom
fix/false-absences-17

Conversation

@markramm

Copy link
Copy Markdown
Owner

The false-absences theme: never certify an absence over a search that wasn't complete. It adds #17, which is the same invariant, and keeps raw friction logs out of the public repo.

False absences fixed (each with a regression test)

  • AWS WAF soft block: it was read as a document. It is now typed AWS_WAF, which is browser-escalatable. Detection keys on response headers plus no visible prose.
  • usaspending.dollar_sum: award rows whose amounts wouldn't parse were certified as "no awards" (exit 11). That is now an AccessBlocker, and the zero-rows-but-hasNext case is fixed too.
  • Cache replay: a cached absence lost its probes, not_searched and caveats. Scope now persists in a sidecar.
  • Sweep test: tests/test_every_source_absence.py runs every source entry point against transport failure, rate limiting and a malformed payload. oscn and docs are xfail (HTML scrapers; tracked).
  • Usaspending certifies a false absence for an IDV PIID #17, counts():
    • It now lists every award group it skipped in not_searched.
    • A PIID-shaped query routes by its 9th character. A/B/D (IDV instruments) widen the count to include IDVs, with a caveat.
    • Q/R mark a solicitation: counts(), search() and dollar_sum() return WRONG_ID_TYPE pointing at SAM.gov before spending a ledger slot. Never an absence.
    • Regression test for 70CMSW26D00000016.

Browser allow-list

  • Operators extend it in ~/.evidence-search/allowed_hosts.json, and a reason is required per host. The shipped baseline can't be reduced, and evidence-search hosts shows the current list.
  • Adds Sunbiz, Kentucky SOS, Schuylkill County PA and Berks County PA. All are public business or property registries.

Feedback hygiene

  • Triage: FEEDBACK.md gets a triage pass: [tracked] markers on the two recurring clusters, a corrected stale header, and a 2026-09-19 summary.
  • Split files: raw entries now go to the gitignored FEEDBACK.local.md. The tracked FEEDBACK.md takes redacted entries only, and every existing entry has been redacted to that standard.
  • Skill: the usage skill is updated to match.
  • Tests: they fail on a local path in FEEDBACK.md or on the local file dropping out of .gitignore. They also tripwire an untriaged backlog over 40.

Review notes

  • Cherry-picked: the theme's commits from the local fix/false-absences branch were cherry-picked onto origin/main. That branch sat on unpushed local-main friction logs that name private research, and those are not published here.
  • Reworked on the way: the triage commit's FEEDBACK changes were rebuilt against the published file. A 2026-09-20 field report (a docket-watch that found a PDF-bucket 404 is not a docket-entry absence, and that --type r lacks total_matches) is included in generalized form.
  • Follow-ons to Usaspending certifies a false absence for an IDV PIID #17, not in this PR: --idv <PIID> child awards; --agency; a reporting-lag note. A listing (search) of a D-shaped PIID with the default --group contracts still lists contracts only. Its not_searched names the other groups, but it doesn't reroute.

Tests: ./.venv/bin/python -m pytest -q gives 262 passed, 2 xfailed.

Fixes #17

🤖 Generated with Claude Code

markramm and others added 9 commits September 29, 2026 10:49
CourtListener began answering non-browser clients with an AWS WAF
JS-challenge: HTTP 202, an empty body, and `x-amzn-waf-action: challenge`.
The challenge names no vendor in its markup, so every _SIGNATURES pattern
missed it -- and fetch() discarded response headers entirely, so the one
place the mechanism WAS named never reached detect_blocker.

The empty body therefore flowed downstream as content, and `extract`
printed "the document was read, the terms are not in it" over a fetch that
read nothing. That is a false negative manufactured by the tool, which is
the single failure this package exists to prevent.

Reported 2026-08-28 with a correct curl root-cause and a written spec, and
again on 2026-09-17 by a worker who re-derived the whole diagnosis because
nothing marked the first report as known. The second reporter noted the
only tell was a `raw ~1 tok` line they happened to have a successful
`raw ~54,024 tok` fetch above in scrollback to compare against.

detect_blocker now reads headers and classifies the pair (challenge-shaped
response, no visible prose) as AWS_WAF, which is browser-escalatable and
names the retry in its detail. Detection is never on size alone: a 202
carrying real prose is served, and a legitimately tiny page stays a Hit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dollar_sum counted only rows whose `Award Amount` was numeric, then
reported "no priced rows" as a VerifiedAbsence at exit 11 -- which a shell
caller reads as publishable. `n == 0` conflated two different facts: the
recipient has no awards, and the recipient has awards we failed to parse.

Feeding two real awards whose amounts arrived as strings ("5,000,000" -- a
shape this module already defends against in _parse_counts) certified a
$7.5M vendor as having no federal awards, with claim() == "Not found".

The Coverage guard could not catch it. That guard refuses coverage a
source DECLARES dirty; here the source built a clean Coverage over a
failure it never noticed. dollar_sum is also the one entry point that
re-implements the run_source shell by hand, which is why it carried a bug
every shell-routed source was structurally protected from.

Unparseable rows and an unread `hasNext` page are now AccessBlocker. A
genuine absence still certifies, and now carries probes plus the subaward
caveat instead of the bare, unauditable "Not found".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cache stored only result rows, so replay_cached rebuilt a
VerifiedAbsence without its probes, not_searched or caveats. claim()
degraded from a scoped sentence to the bare string "Not found" -- same
question minutes apart, same exit 11, materially weaker claim.

That is the opposite of the README's own worked example, where the
published sentence survives a lawyer precisely BECAUSE the tool printed
the subaward caveat: "without it the sentence is an overclaim a lawyer
could take apart." The tool produced the parenthetical on a cache miss and
dropped it on a cache hit, within a 24h TTL (7 days for crossref).

Severity was bounded -- is_absolute requires bool(probes), so a scopeless
replay could never read as an absolute negative -- but the caveats are the
part a reporter copies into the piece.

Scope now persists in a `__rows__`/`__meta__` sidecar rather than a schema
change, so rows written by an earlier version still read as bare lists and
simply replay the weaker claim they were stored with. Staleness disclosure
is unchanged: a replayed absence still says when it was established.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-source absence tests are hand-written spot checks, and they are
only as complete as the list someone thought to write. That is how
usaspending.dollar_sum shipped a false absence with 209 tests green:
nobody spot-checked the one function that re-implements the shell by hand,
so nothing enforced the invariant there at all.

This walks EVERY registered entry point against transport failure, rate
limiting, and a malformed success payload, asserting that none yields a
VerifiedAbsence. The point is the next source: a contributor who adds one
and forgets its absence test still gets the sweep, and one who hand-rolls
their own Coverage -- the thing that caused two of the three live bugs --
fails here.

Two entry points are xfail rather than hidden. oscn.search and docs.search
scrape HTML, and a scraper counting rows genuinely cannot tell "the page
served no results" from "this is not the page I know." Fixing that needs a
positive shape check of the kind oscn already has for its real empty page.
Tracked, not fixed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ALLOWED_HOSTS was a hardcoded set of 18 with no override, so reaching any
new government registry required a code commit and a release. Agents filed
46 reports naming hosts they could not reach -- loc.gov (12 mentions),
fec.gov (9), justice.gov (5), apps.occ.gov, scgov.net, county clerks --
while the error message told them to "add it to ALLOWED_HOSTS
deliberately" and gave no route to do so. Two reports asked for the list
to be queryable. Meanwhile the two hosts that did get added appear nowhere
in FEEDBACK.md, so the only code changes in a month were not the ones
being asked for.

A hardcoded set is the right default and the wrong only option. Operators
now extend it in ~/.evidence-search/allowed_hosts.json (or
$EVIDENCE_ALLOWED_HOSTS), and `evidence-search hosts` prints the current
boundary.

The ethical boundary is unchanged and tested. A reason is REQUIRED per
host -- an entry without one is refused, so the file cannot decay into an
undocumented pile of hostnames, and CONTRIBUTING's "adding a host is a
deliberate act" still holds. The match is still the dot: allowing
sos.example.gov never allows evil-sos.example.gov. A malformed config
fails closed to the shipped baseline, which config can extend but never
reduce.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CONTRIBUTING claims field reports drive fixes better than synthetic tests.
That was true, and the seed-era triage proves it -- but the loop ran
one-directional for a month: reports kept arriving and almost none became
code or tests.

Every [fixed]/[wontfix]/[tracked] marker sat in the seed era while entries
accumulated untriaged, and the file's own header went on claiming
everything below it was marked. The cost is measurable: the CourtListener
soft block was reported with a correct root cause and a written spec, went
unmarked, and was re-derived from scratch weeks later by a worker who paid
the whole diagnostic cost again.

Triages the two recurring clusters (web rate limiting, browser allow-list
gaps), corrects the stale header, and appends the triage pass. Original
entry text is left intact, per the file's own rule.

Adds tests/test_feedback_hygiene.py so this is mechanical rather than
remembered: one test tripwires when the untriaged backlog passes 40, the
other fails if the file asserts a triage it has not done.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
)

`usaspending "70CMSW26D00000016" --keywords --count` exited 11 with a
VerifiedAbsence. The award exists, as an IDV; `--all-types` finds it. The
CLI counts contracts only by default, and counts() never said which award
groups it had skipped, so a zero over contracts read as a zero over every
federal award.

Same invariant as the rest of this branch: a negative must state its own
scope.

- counts() lists every award group it did not search in not_searched.
- A PIID-shaped query routes by its 9th character, the instrument type:
  A/B/D (BPA, BOA, indefinite-delivery) are IDVs, so a count that left IDVs
  out now includes them, with a caveat saying so. Widening a count can find
  more but can never manufacture an absence.
- Q/R mark a solicitation, which USAspending cannot hold. counts(), search()
  and dollar_sum() return a WRONG_ID_TYPE AccessBlocker pointing at SAM.gov
  before spending a ledger slot, never an absence.

Regression tests in tests/test_every_source_absence.py, including the live
70CMSW26D00000016 case.

Fixes #17

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Florida Division of Corporations (Sunbiz) serves a Cloudflare interstitial
to a plain fetch; Kentucky SOS business search is an ASP.NET postback form;
Schuylkill County PA and Berks County PA publish real-property records
behind JS-rendered search pages. All four are government public-records
systems -- business registries and property records -- the category the
allow-list exists for. Each entry carries its reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FEEDBACK.md is tracked in a public repository, and agents write their field
reports mid-task -- naturally naming the task, the subject, the ticket and
their working paths. That is the right raw material and the wrong thing to
publish.

- FEEDBACK.local.md is gitignored and is where raw entries now go.
- FEEDBACK.md takes redacted entries only, and says so in its header:
  people, investigations, vendors, committees and places become stable
  placeholders; tasks are described by kind, not ticket name; no local
  paths, KB ids, ticket or draft names. Public identifiers that are the bug
  (PIIDs, endpoints, government domains) stay. Every existing entry has
  been redacted to that standard; the report text is otherwise unchanged.
- The evidence-search-usage skill tells agents to log raw to the local file
  and redact before anything reaches FEEDBACK.md or an issue.
- tests/test_feedback_hygiene.py fails on a local path in FEEDBACK.md and
  on FEEDBACK.local.md dropping out of .gitignore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@markramm
markramm marked this pull request as ready for review September 29, 2026 14:59
@markramm
markramm merged commit 2c5342e into main Sep 29, 2026
3 checks passed
@markramm
markramm deleted the fix/false-absences-17 branch September 29, 2026 14:59
markramm added a commit that referenced this pull request Sep 29, 2026
…d keyless, issue numbers; add www.gao.gov to the browser allow-list

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Usaspending certifies a false absence for an IDV PIID

1 participant