One-command reproduction and the funnel figure (#25)

.github/workflows/ci.yml

@@ -23,6 +23,22 @@ jobs:
       - name: Type-check
         run: uv run --frozen mypy
 
+  reproduce:
+    # Every committed benchmark table, summary and figure, regenerated offline and compared with the repository.
+    timeout-minutes: 20
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - name: Install locked tooling
+        run: python -m pip install uv==0.12.17
+      - name: Install locked project
+        run: uv sync --frozen --extra dev
+      - name: Reproduce every result and figure
+        run: uv run --frozen --with matplotlib==3.11.2 python tools/reproduce.py
+
   verify:
     timeout-minutes: 15
     strategy:

CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ## Unreleased
 
+- Add `tools/reproduce.py`: one command regenerates every committed benchmark table, summary and figure
+  offline (18 steps, about three minutes) and compares each with the repository; CI runs it on every
+  change. It caught the ClinVar and VBO figures still captioned v0.7.0 after the release, now fixed.
+- Add the funnel figure (`docs/assets/ai_validation_funnel.svg`): single-cell held-out answers, stage
+  by stage, from the first check through feedback to what was admitted and how it compares with the
+  authors' terms. Cell-type summaries now record these stages.
 ## 0.8.0 — AI validation: grounding, generic grounders, the feedback loop and benchmarks
 
 Bioevidence now checks AI output against pinned sources and reference releases, feeds fixable

README.md

@@ -95,6 +95,10 @@ The same loop corrects identifiers in single-cell annotation. Claude Haiku 4.5 l
 
 Its second answer, `CL:0000173`, was admitted and is exactly the authors' annotation.
 
+Across all 276 held-out annotations, stage by stage:
+
+![Funnel for the single-cell held-out clusters: of 276 first answers, 198 pass every check, 63 have a wrong identifier or marker and are sent back, 15 contradict themselves and go to a person; after feedback 255 are admitted and 21 go to a person; of the admitted, 205 agree with the authors' term and 50 disagree; the expert audit of admitted records is not yet run](https://raw.githubusercontent.com/NingyuSUN/bioai-evidence-validator/main/docs/assets/ai_validation_funnel.svg)
+
 **Failure handling**, each in the code and exercised in the runs:
 
 | Failure | Detected by | Handling |
@@ -240,6 +244,24 @@ More: [draft format](https://github.com/NingyuSUN/bioai-evidence-validator/blob/
 
 ![ClinVar benchmark: share of 2023 pathogenic classifications reclassified or conflicting by 2026, with and without a dissenting submission, and false admissions under controlled faults and the trust boundary](https://raw.githubusercontent.com/NingyuSUN/bioai-evidence-validator/main/docs/assets/clinvar_germline_benchmark.svg)
 
+## Reproduce every number
+
+```bash
+uv run --frozen --with matplotlib==3.11.2 python tools/reproduce.py
+```
+
+One command regenerates every committed benchmark table, summary and figure, offline, in about three minutes,
+and compares each with the repository byte for byte (text files with line endings normalised). It runs 18 steps,
+also run in CI on every change:
+- the three real-data cases;
+- ten LLM-benchmark result sets;
+- the task files and the error taxonomy;
+- the figures.
+
+Model calls, agent runs and downloads are not repeated. What they produced is committed and pinned by
+SHA-256: model answers, agent episodes, the verification of cited papers, and source snapshots. `--write` updates the
+committed files and `--list` shows the steps.
+
 ## Limits
 
 - **Admission is not truth.** It means the record meets the selected profile: its evidence exists in the
@@ -256,8 +278,8 @@ More: [draft format](https://github.com/NingyuSUN/bioai-evidence-validator/blob/
 Tracked in the [AI validation roadmap](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/AI_VALIDATION_ROADMAP.md) (#26):
 - expert review of the benchmark cases (#23);
 - calibrated triage and audit sampling of admitted records, to measure what still gets through (#24);
-- one-command reproduction of every benchmark, a funnel figure, and a validation dossier for release
-  0.8.0 (#25).
+- a validation dossier structured like FDA's draft AI credibility framework (#25). One-command reproduction
+  and the funnel figure are done.
 
 ## Contributing and citing
 

docs/ENGINEERING.md

@@ -297,8 +297,18 @@ uv sync --frozen --extra dev
 uv run --frozen pytest
 uv build
 uv run --isolated --no-project --with ./dist/*.whl python tools/check_distribution.py
+uv run --frozen --with matplotlib==3.11.2 python tools/reproduce.py   # every benchmark result and figure
 ```
 
+`tools/reproduce.py` regenerates every committed benchmark table, summary and figure offline from what is
+committed, and compares each with the repository: text with line endings normalised, everything else byte for
+byte. What it replays from:
+- for the real-data cases, the hash-checked source snapshots;
+- for the LLM benchmark, the recorded model answers, agent episodes and paper verifications;
+- for the figures, the summaries, with matplotlib pinned.
+
+CI runs it on every change, and a test fails if a results directory or figure is not covered by one of its steps.
+
 CI runs on Linux/Python 3.11–3.14 and Windows/Python 3.13, and runs the GitHub Action on
 Linux and Windows. Tests cover multi-domain acceptance, negative evidence cases,
 use-specific human review, strict configuration/JSON/draft parsing, baseline enforcement,