[timodonnell]

Summary

Tracks #336.

• freezes and reconciles the current exp: retrain our best model on decontaminated training data #232 AFDB + ESM-Atlas universe (69,516,181 documents / 74,475,864,003 source tokens)
• implements a deterministic direct-witness sequence/structure selector that cannot remove through transitive chaining
• runs a full 50%-identity, 80%-bidirectional-coverage sequence candidate pass and explicit Smith-Waterman verification
• derives comparable 50/70/90/95/100% sequence-only ceilings
• calibrates TM thresholds from the pinned exp: expand AFDB and ESMFold2 training with structurally diverse cluster members #292 measured-pair gallery
• records compact CSV/provenance artifacts, plots, and a living summary PDF

Current result

At 50% identity, the direct-witness sequence-only pass removes 5,022,836 / 69,516,181 documents (7.225%) and retains 64,493,345. That decomposes into 667,093 AFDB removals (16.833% of AFDB) and 4,355,743 ESM removals (6.645% of ESM). The exact AFDB token loss is 738,657,274 / 4,432,940,838 (16.663%).

This is an upper bound on removals under the joint rule: TM/contact evidence may rescue rows. It is candidate-limited because Linclust is not an exhaustive all-pairs graph.

Still outstanding before this leaves draft

• exact ESM token accounting from all 3,338 source-local shards
• stratified source-local structure/contact smoke and cost projection
• corpus-wide 50% / TM-0.7 estimate (or a sampled estimate with uncertainty if full scoring is not justified)
• 40% and 30% candidate passes

Validation

• uv run pytest -q — 27 passed
• uv run ruff check . — clean
• uv run python plot_pilots.py && uv run python build_summary.py