Skip to content

Add lda and fec sources: LDA bill search and FEC bulk committee transactions - #23

Draft
markramm wants to merge 1 commit into
mainfrom
feature/lda-fec-sources
Draft

markramm wants to merge 1 commit into
mainfrom
feature/lda-fec-sources

Conversation

@markramm

@markramm markramm commented Oct 1, 2026

Copy link
Copy Markdown
Owner

What

Theme A (#19, LDA part only). A keyless lda source over the Senate LDA API (https://lda.gov/api/v1/; lda.senate.gov 301-redirects there, verified 2026-09-29). Queries by client and/or registrant and year(s); --bill "H.R. 2994" filters client-side by scanning each filing's lobbying_activities[].description, normalizing spacing/periods ("H.R. 2994", "H.R.2994", "HR 2994" all match). A scoped VerifiedAbsence names what wasn't searched: filings before 1999 (the API's coverage start), LD-203, paper-era filings, and unreconciled amendments. Congressional financial disclosures are left as the remainder of #19.

Theme B (#16). A bulk-first fec source. Downloads FEC's pas2/oth/cm bulk files into a local SQLite index (gitignored cache dir, default ~/.evidence-search/fec_bulk, override EVIDENCE_FEC_CACHE), and answers "committee X → committee Y in cycles C1-C2" with both sides shown where both reported, refunds netted (shown, never dropped), image numbers and transaction IDs, and a committee-name lookup that warns when a name matches more than one committee ID.

A real bug the regression fixture caught

Building the fixture against real bulk data (Purdue Pharma PAC C00370643 → the two Mike Rogers committees and a leadership PAC) surfaced something docs/extension-candidates.md's design note didn't anticipate: FEC's pas2 and oth bulk extracts are not disjoint for committee-to-candidate rows. The same schedule line — identical SUB_ID, image number and transaction ID — appears in both files. Summing both put five real $1,000 contributions at $10,000. _dedupe_by_sub_id fixes it, and the fixture includes the real duplicate rows (not just the ones I originally assumed were the whole story) so the dedup path is actually exercised, per CONTRIBUTING's point about dollar_sum shipping green without a test that would have caught its bug.

Also: a cold index of 3 cycles (~2M real rows) took minutes with per-row INSERT on default SQLite settings; switched to executemany + WAL + synchronous=NORMAL (the same tradeoff Store() already makes), which brought it to seconds.

Regression results

LDA — real, trimmed API response (tests/fixtures_lda_purdue_2008.json): Purdue Pharma LP's 2008 Q1 and Q2 LD-2 filings both name "National Pain Care Policy Act (H.R. 2994)" — Hit, 2 results, verified against the live API as well (which showed Q3/Q4 also name it, a detail the issue's summary didn't claim either way).

FEC — real bulk-file rows extracted via grep/awk (tests/fixtures_fec_purdue_rogers.py), no downloads in the test suite: Purdue Pharma PAC C00370643 → Rogers for Congress C00343863 is 5 × $1,000 across 2005-2010; → Mike Rogers for Congress (AL) C00367862 is a $1,000 contribution (2007-10-09) netted against a -$1,000 refund (2007-12-31) to $0, both rows shown; → the leadership PAC C00370791 shows both sides independently reporting the same $1,000 in 2006 and 2012. Committee-name lookup on "Rogers" correctly warns across the two distinct IDs.

Both sources follow the existing run_source/absence-invariant conventions, declare corpus/not_searched/caveats, are added to tests/test_every_source_absence.py's generic sweep (transport failure, rate-limited, malformed payload — never an absence), and are wired into the CLI with dedicated dispatch tests.

Commands run

git -C ~/evidence-search worktree add ../evidence-search-wt-sources -b feature/lda-fec-sources origin/main
./.venv/bin/pytest -q                                   # 322 passed, 2 xfailed
evidence-search lda --client "Purdue Pharma" --year 2008 --bill "H.R. 2994"   # live, Hit
evidence-search fec --from-committee C00370643 --to-committee C00343863 --cycles 2006-2010  # live, Hit

What remains

  • Re-probe LDA and congressional financial disclosures #19: congressional financial disclosures (House Clerk, Senate eFD) — not probed or built here.
  • FEC campaign-finance source, bulk first #16: an optional OpenFEC API path for recency beyond the latest bulk file, per the issue's "optional Tier 2 API path" note.
  • Field friction (the pas2/oth overlap finding, local-index size/perf, nothing story-specific) is logged in FEEDBACK.md; raw session notes including an unrelated low-disk-space condition on the dev machine are in the gitignored FEEDBACK.local.md.

Refs #16, #19.

🤖 Generated with Claude Code

…actions

Theme A (#19, LDA part): a keyless lda source over the Senate LDA API
(lda.gov/api/v1; lda.senate.gov 301-redirects there). Queries by client
and/or registrant and year(s); --bill "H.R. 2994" scans each filing's
lobbying_activities descriptions for a bill number, normalizing spacing and
periods. A scoped VerifiedAbsence names what wasn't searched: pre-1999
filings, LD-203, paper-era filings, and unreconciled amendments. Regression
test uses a real, trimmed API response (Purdue Pharma LP's 2008 Q1/Q2 LD-2s
naming H.R. 2994). Congressional financial disclosures are left for the rest
of #19.

Theme B (#16): a bulk-first fec source. Downloads FEC's pas2/oth/cm bulk
files into a local SQLite index (gitignored cache, default
~/.evidence-search/fec_bulk), answers "committee X -> committee Y in cycles
C1-C2" with both sides shown where both reported, refunds netted (never
dropped), and a committee-name lookup that warns on multiple matching IDs.

Building the regression fixture against real bulk data (Purdue Pharma PAC
C00370643 -> two Mike Rogers committees and a leadership PAC) surfaced a real
bug: FEC's pas2 and oth bulk extracts are NOT disjoint for committee-to-
candidate rows -- the same schedule line (same SUB_ID) appears in both,
which double-counted every contribution before _dedupe_by_sub_id was added.
The fixture includes the real duplicate rows so the dedup path is actually
exercised. Also switched the local index to executemany + WAL (a cold
3-cycle index of ~2M real rows went from minutes to seconds).

Both sources follow the existing run_source/absence-invariant conventions,
are added to tests/test_every_source_absence.py's generic sweep, and are
wired into the CLI with argparse + dispatch tests. Full suite: 322 passed,
2 xfailed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant