Merge pull request #17 from NingyuSUN/model-reviewers

Model reviewers on the blinded ClinVar packet, and their analysis (#17, #22)

Author: NingyuSUN

CHANGELOG.md

@@ -2,6 +2,14 @@
 
 ## Unreleased
 
+- Add model reviewers for the blinded ClinVar review packet (#17, #22). `model_reviewers.py` gives the six benchmark
+  models exactly what experts get, one isolated session per case, with tools disabled or detected, stops a model
+  whose account is out of quota, and lets Claude run on a different install than Codex and Antigravity. All six
+  reviewed all 190 cases with no tool use. `analyze_models.py` reports inter-model agreement now (alpha 0.49–0.53
+  for the clinical and expert uses: the models often split), and, once the expert labels exist, scores per subset,
+  error correlation, whether unanimous models are right, and a decision rule fixed before any label. Per-case model
+  labels stay private until the expert review is complete; the published summary holds only what cannot unblind a
+  reviewer.
 - Add the real extraction experiment (#21, `examples/civic_extraction`, `evaluation/llm_benchmark/extraction_loop.py`).
   Six models extract clinical evidence claims on their own from 50 openly licensed papers that CIViC curated (10
   pilot, 40 test; protocol frozen before the test split). Each claim, written in the fields of `draft-schema`,

README.md

@@ -290,6 +290,8 @@ context of use, model risk, credibility evidence, adequacy): the
 
 Tracked in the [AI validation roadmap](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/AI_VALIDATION_ROADMAP.md) (#26):
 - expert review of the benchmark cases (#23), including the 60-claim sample from the extraction experiment (#21);
+- scoring the six models' reviews of the 190 ClinVar cases against those expert labels. The models have reviewed
+  them, and they often disagree with each other (#22);
 - an expert audit of admitted records with `bioevidence review audit-sample` and `audit-score`, which bound
   the error rate of what still gets through. The tool and a dry run are done (#24); the audit needs experts.
 - the [validation dossier](https://github.com/NingyuSUN/bioai-evidence-validator/blob/main/docs/VALIDATION_DOSSIER.md),

docs/AI_VALIDATION_ROADMAP.md

@@ -49,7 +49,7 @@ source → structured extraction → provenance → deterministic validation
 | Structured extraction | Did the AI turn the source into the right claim? | Draft format and per-profile JSON Schema for LLM output | Run with six models on 40 held-out openly licensed papers: no admitted claim with a wrong identifier, name or quote; semantic correctness awaits expert labels | [#21](https://github.com/NingyuSUN/bioai-evidence-validator/issues/21) |
 | Provenance | Can each item be traced to its source and method? | Source artifacts, locators, methods; PROV/ECO/VA-Spec mapping; locators and review tiers recomputed from pinned snapshots | Methods are self-declared | Signed attestations (future) |
 | Deterministic validation | Is the evidence sufficient for this use, by fixed rules? | Engine, profiles, per-type quality gates, audit reports | — | — |
-| Model/agent evaluation | What can model review replace, and where does it fail? | Blinded model-review runner (#17), pilot on 20 cases | No reference labels yet; error correlation unmeasured | [#22](https://github.com/NingyuSUN/bioai-evidence-validator/issues/22) |
+| Model/agent evaluation | What can model review replace, and where does it fail? | Blinded model-review runner: six models reviewed all 190 cases; the analysis (scores, error correlation, a decision rule fixed in advance) is ready | Agreement among the models is moderate (alpha 0.49–0.53 for the clinical and expert uses); correctness needs the expert labels | [#22](https://github.com/NingyuSUN/bioai-evidence-validator/issues/22) |
 | Trust boundary | What does the system still take on faith? | Source grounding recomputes identifiers, review tiers, completeness, retractions and quotes from pinned snapshots (0/80 structured-source forgeries and 0/489 literature controls admitted) | Whether a verbatim quote supports the claim | [#31](https://github.com/NingyuSUN/bioai-evidence-validator/issues/31) |
 | Human review | What do experts conclude, and how reliably? | Protocol, `bioevidence review` tooling, blinded ClinVar kit | **No expert labels yet** | [#23](https://github.com/NingyuSUN/bioai-evidence-validator/issues/23) |
 
@@ -81,7 +81,7 @@ source → structured extraction → provenance → deterministic validation
 | Done | [#20](https://github.com/NingyuSUN/bioai-evidence-validator/issues/20) | Source grounding, phase 2: verify cited literature and quotes | #19 |
 | Now | [#31](https://github.com/NingyuSUN/bioai-evidence-validator/issues/31) | Semantic support check: does a verbatim quote support the claim's direction? | #20 |
 | Done | [#21](https://github.com/NingyuSUN/bioai-evidence-validator/issues/21) | Real AI extraction experiment on openly licensed sources (expert labels of the sample need #23) | #18, #20 |
-| Next | [#22](https://github.com/NingyuSUN/bioai-evidence-validator/issues/22) | Model and agent evaluation against expert labels | #17, #23 |
+| Now | [#22](https://github.com/NingyuSUN/bioai-evidence-validator/issues/22) | Model and agent evaluation against expert labels (models reviewed; scoring waits for #23) | #17, #23 |
 | Done | [#24](https://github.com/NingyuSUN/bioai-evidence-validator/issues/24) | Calibrated triage and audit of auto-admitted records (tooling, routing evidence and a dry run; the expert audit needs #23) | #23 |
 | Done | [#25](https://github.com/NingyuSUN/bioai-evidence-validator/issues/25) | Reproducible benchmark, funnel figure and validation dossier | all |
 

evaluation/clinvar_review/README.md

@@ -113,6 +113,103 @@ Then commit `annotations.csv`, `adjudications.csv`, `agreement.json`, `score.jso
 `manifest.json` and the key into `evaluation/clinvar_review/results/`, and update the ClinVar
 case README, which currently states that no independent expert annotation exists.
 
+## Model reviewers (#22)
+
+The same packet goes to the six models of the LLM benchmark, to measure which parts of expert review they could
+take over. `model_reviewers.py` drives three command-line agents: Claude Code (`claude`), Codex (`codex`) and
+Antigravity (`agy`, for Gemini). Each vendor contributes a frontier model and a fast one:
+- Claude Opus 5.5 and Claude Haiku 4.5;
+- GPT-6-Astra and GPT-5.6-Luna;
+- Gemini 3.1 Pro and Gemini 3.8 Flash.
+
+Each model gets exactly what a human reviewer gets, the rubric and one case under its code, and answers through a
+JSON schema.
+
+```bash
+# Where the CLIs are installed; reads only reviewer_packet/, never the key
+python3 evaluation/clinvar_review/model_reviewers.py run \
+  --packet artifacts/clinvar-review/reviewer_packet --output artifacts/clinvar-review/models \
+  --set calibration                     # start small; --limit N, --codes, --reviewers
+
+# Maintainer, with the key: protocol annotations and predictions, then the analysis
+uv run --frozen python evaluation/clinvar_review/model_reviewers.py export \
+  --output artifacts/clinvar-review/models --key artifacts/clinvar-review/maintainer/key.json
+uv run --frozen python evaluation/clinvar_review/analyze_models.py \
+  --models artifacts/clinvar-review/models --output artifacts/clinvar-review/models/analysis
+```
+
+- **Isolation.** One case per call. Every call is a fresh, non-interactive session in an empty temporary
+  directory, so the model cannot read the key or other cases.
+- **No lookups.**
+  - Claude Code runs with every tool disabled.
+  - Codex runs without browser tools or your user config and MCP servers, in a read-only sandbox.
+  - Antigravity cannot switch off web search. Its tool calls are detected from the event stream.
+  - A case answered after a tool call is retried once. If a tool is used again, the case is kept but marked
+    `later_information_seen = yes`.
+- **Quota.** A CLI whose account is out of quota stops that reviewer; nothing is recorded for its remaining cases,
+  and the next run resumes them. The Claude models share one call slot. Several installs can write to one output
+  folder (for example Claude on Windows and Codex and Antigravity in WSL); `manifest.json` records the CLI version
+  and path that answered for each model.
+- **Record.** `manifest.json` records model IDs, CLI versions and hashes of the rubric, prompt and schema. `raw/`
+  keeps every answer. `runs.csv` lists time, tokens and tool use per call. Re-running resumes.
+
+### What is published, and when
+
+Model labels could steer a human reviewer, so they follow the same rule as the key: **per-case model labels stay
+private until all expert reviews are in.** So does anything that compares models with the key, the validator or
+NCBI's review status. Until then `analyze_models.py --public` writes only:
+- how each model ran: answers, tool use, later information seen, time;
+- how much the models agree with each other, per use (Krippendorff's alpha, bootstrap interval);
+- the decision rule below;
+- the SHA-256 of the withheld per-case files.
+
+The published summary is in [`results/models`](results/models/summary.md).
+
+### Results so far
+
+All six models reviewed all 190 cases.
+- **Clean runs.** No model used a tool or said it recognised later information.
+- **Two runs.** The calibration round (20 cases) ran on 2026-09-28; the rest ran on 2026-10-08. GPT-6-Astra's
+  Codex CLI was updated in between, from 0.153.4 to 0.160.0.
+- **Two accounts' quotas.** Claude ran on the Windows install's account. Antigravity's account ran out of quota
+  partway through, and its 35 unanswered cases were rerun after the reset.
+
+On the 170 test cases, before any expert label:
+
+| Label | All six agree | Krippendorff's alpha (95% CI) |
+|---|---:|---|
+| Research summary | 91.8% | 0.12 (0.03–0.21) |
+| Clinical reference | 55.3% | 0.53 (0.43–0.62) |
+| Expert reference | 38.2% | 0.49 (0.43–0.55) |
+| Statement correct? | 86.5% | 0.47 (0.24–0.62) |
+
+- **High raw agreement means little where nearly every case gets one label.** For research summaries the six
+  models agree on 92% of cases, but alpha is close to chance.
+- **For the uses that depend on the evidence, the models often split.** All six agree on about half the cases or
+  fewer.
+
+So their consensus cannot stand in for an expert's decision. Whether any of them is *right* is what the expert
+labels will show (#23).
+
+### Analysis against the expert labels
+
+Once the expert labels are resolved, `analyze_models.py --annotations … --adjudications … --predictions
+maintainer/predictions.csv` adds, on the test split:
+- **Scores** for each model, the models' majority (at least four of six), the validator and NCBI's review status,
+  per use and per subset (divergent, control).
+- **Error correlation.** For each pair of models: their misses, the misses they share, P(B misses | A misses), and
+  Cohen's kappa of the miss indicators. Shared misses make a second model's opinion worth little.
+- **Does agreement mean correctness?** How often the six models are wrong when they all agree, and when they split.
+- **Decision statement, by a rule fixed before any expert label exists.** Per use:
+  - **Models alone**, if the models' majority wrongly admits cases that experts would not admit at a rate whose
+    exact one-sided 95% upper bound is below 5%, and wrongly blocks fewer than 20% of the cases experts would
+    admit, on the divergent and the control subset each.
+  - **Deterministic rules**, if the validator meets the same two conditions.
+  - **Experts** otherwise.
+
+Model labels are **not independent human annotations**. Keep them in their own files, never in `annotations.csv`,
+and never use them to resolve an expert disagreement.
+
 ## Interpreting the result
 
 - With two independent reviewers and adjudication this is an **independently reviewed