Repository navigation
Conversation
…actions Theme A (#19, LDA part): a keyless lda source over the Senate LDA API (lda.gov/api/v1; lda.senate.gov 301-redirects there). Queries by client and/or registrant and year(s); --bill "H.R. 2994" scans each filing's lobbying_activities descriptions for a bill number, normalizing spacing and periods. A scoped VerifiedAbsence names what wasn't searched: pre-1999 filings, LD-203, paper-era filings, and unreconciled amendments. Regression test uses a real, trimmed API response (Purdue Pharma LP's 2008 Q1/Q2 LD-2s naming H.R. 2994). Congressional financial disclosures are left for the rest of #19. Theme B (#16): a bulk-first fec source. Downloads FEC's pas2/oth/cm bulk files into a local SQLite index (gitignored cache, default ~/.evidence-search/fec_bulk), answers "committee X -> committee Y in cycles C1-C2" with both sides shown where both reported, refunds netted (never dropped), and a committee-name lookup that warns on multiple matching IDs. Building the regression fixture against real bulk data (Purdue Pharma PAC C00370643 -> two Mike Rogers committees and a leadership PAC) surfaced a real bug: FEC's pas2 and oth bulk extracts are NOT disjoint for committee-to- candidate rows -- the same schedule line (same SUB_ID) appears in both, which double-counted every contribution before _dedupe_by_sub_id was added. The fixture includes the real duplicate rows so the dedup path is actually exercised. Also switched the local index to executemany + WAL (a cold 3-cycle index of ~2M real rows went from minutes to seconds). Both sources follow the existing run_source/absence-invariant conventions, are added to tests/test_every_source_absence.py's generic sweep, and are wired into the CLI with argparse + dispatch tests. Full suite: 322 passed, 2 xfailed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Theme A (#19, LDA part only). A keyless
ldasource over the Senate LDA API (https://lda.gov/api/v1/;lda.senate.gov301-redirects there, verified 2026-09-29). Queries by client and/or registrant and year(s);--bill "H.R. 2994"filters client-side by scanning each filing'slobbying_activities[].description, normalizing spacing/periods ("H.R. 2994", "H.R.2994", "HR 2994" all match). A scopedVerifiedAbsencenames what wasn't searched: filings before 1999 (the API's coverage start), LD-203, paper-era filings, and unreconciled amendments. Congressional financial disclosures are left as the remainder of #19.Theme B (#16). A bulk-first
fecsource. Downloads FEC'spas2/oth/cmbulk files into a local SQLite index (gitignored cache dir, default~/.evidence-search/fec_bulk, overrideEVIDENCE_FEC_CACHE), and answers "committee X → committee Y in cycles C1-C2" with both sides shown where both reported, refunds netted (shown, never dropped), image numbers and transaction IDs, and a committee-name lookup that warns when a name matches more than one committee ID.A real bug the regression fixture caught
Building the fixture against real bulk data (Purdue Pharma PAC
C00370643→ the two Mike Rogers committees and a leadership PAC) surfaced somethingdocs/extension-candidates.md's design note didn't anticipate: FEC'spas2andothbulk extracts are not disjoint for committee-to-candidate rows. The same schedule line — identicalSUB_ID, image number and transaction ID — appears in both files. Summing both put five real $1,000 contributions at $10,000._dedupe_by_sub_idfixes it, and the fixture includes the real duplicate rows (not just the ones I originally assumed were the whole story) so the dedup path is actually exercised, per CONTRIBUTING's point aboutdollar_sumshipping green without a test that would have caught its bug.Also: a cold index of 3 cycles (~2M real rows) took minutes with per-row
INSERTon default SQLite settings; switched toexecutemany+ WAL +synchronous=NORMAL(the same tradeoffStore()already makes), which brought it to seconds.Regression results
LDA — real, trimmed API response (
tests/fixtures_lda_purdue_2008.json): Purdue Pharma LP's 2008 Q1 and Q2 LD-2 filings both name "National Pain Care Policy Act (H.R. 2994)" —Hit, 2 results, verified against the live API as well (which showed Q3/Q4 also name it, a detail the issue's summary didn't claim either way).FEC — real bulk-file rows extracted via grep/awk (
tests/fixtures_fec_purdue_rogers.py), no downloads in the test suite: Purdue Pharma PACC00370643→ Rogers for CongressC00343863is 5 × $1,000 across 2005-2010; → Mike Rogers for Congress (AL)C00367862is a $1,000 contribution (2007-10-09) netted against a -$1,000 refund (2007-12-31) to $0, both rows shown; → the leadership PACC00370791shows both sides independently reporting the same $1,000 in 2006 and 2012. Committee-name lookup on "Rogers" correctly warns across the two distinct IDs.Both sources follow the existing
run_source/absence-invariant conventions, declarecorpus/not_searched/caveats, are added totests/test_every_source_absence.py's generic sweep (transport failure, rate-limited, malformed payload — never an absence), and are wired into the CLI with dedicated dispatch tests.Commands run
What remains
FEEDBACK.md; raw session notes including an unrelated low-disk-space condition on the dev machine are in the gitignoredFEEDBACK.local.md.Refs #16, #19.
🤖 Generated with Claude Code