[timodonnell]

Addresses #373 by comparing the pinned full-data default and ProteinMPNN-only checkpoints on all 97 natural eval-val proteins, with 19 designs reported separately.

The experiment saves 440,800 individual rollouts: the original 278,400-sample temperature/prompt study plus 162,400 new samples extending temperature-1 fixed prompts from 300 to 1,000 per model/protein. Original 300-sample files remain unchanged. The report includes diversity/count controls, Neff analyses, extended ensemble/discovery curves, and oracle best@k F1 against mean individual-rollout F1.

On natural proteins, MPNN-only has higher diversity but lower accuracy. At 1,000 samples it has a small true-contact coverage advantage (99.0% versus 98.5%), accompanied by 53% more false contacts. Default versus MPNN-only oracle best@1,000 F1 is 0.5607 versus 0.4628; mean individual F1 is 0.4115 versus 0.3020. Oracle selection uses ground truth. Extra samples improve oracle F1 in both models while ensemble accuracy largely plateaus after 300. Neff support remains specific to the extreme-tier contrast; the continuous interaction is inconclusive.

• Complete report with figures
• 22-page summary PDF
• Public extension artifacts; original raw artifacts remain under v1.

Validation: 13 unit tests and Ruff pass; both extension smoke tests and all 12 extension production shards completed. One initialization failure was restarted with identical code/settings, before any samples were generated. All 162,400 new samples were retained, with zero continuations/caps. Audits verify checkpoint hashes, paired prompt tokens/seeds, exact original-prefix hashes, and all 3,248 original discovery/ensemble rows within 1e-12. All 120 published report/input/table files passed anonymous SHA256 readback. Added inference totals 2.57 GPU-hours, excluding setup and retries.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-30T19:21:43.580105Z	e163932	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[claude]

Claude finished @timodonnell's task in 7m 0s —— View job

---

Review tasks

• Read repo guidelines (AGENTS.md, experiments/AGENTS.md)
• Gather PR diff (26 code/doc files, 3,592 lines; remainder is data/ tables + plots/)
• PR-level checks — branch codex/* carries agent-generated; body uses Refs #373, no issue-closing keyword. Clean.
• 4 parallel review agents (2× AGENTS.md compliance, 2× bugs/logic) — all returned empty
• Validation pass — no candidate findings to validate
• Post review

Coverage notes from the review agents

Compliance. No from __future__ imports; no monkey-patching; no compatibility shims. The only try/except blocks are publish_file.py:17-35 (bounded retry on {429,500,502,503,504}, re-raises everything else and on the final attempt) and publish_file.py:50-59 (404 is the intended answer to the --exists probe) — intentional control flow, not swallowed exceptions. Elsewhere the pipeline is fail-loud (analyze.py:352, worker.py:60-68,102, validate_results.py). Both checkpoints are referenced with run name and step (…-exp277-m2-p06-full-epoch-1.5B/hf/step-266344, …-exp364-mpnn-only-1ep-lr5e-4-cooldown-1.5B/hf/step-186884) — no final/ or latest/. No W&B usage, so the routing and history/runs rules don't apply. No gs:// writes; artifacts go to hf://buckets/open-athena/MarinFold/data/exp373-mpnn-diversity-neff/v1, enforced as a prefix at publish_file.py:47. Dir name/kind/frontmatter issue: 373 all consistent. Committed binaries are PNGs + .meta.json sidecars + summary.pdf (3.4 MB) and small data/ tables (8.4 MB, largest 2.3 MB) — required, not violations. Tests assert exact values rather than relaxed tolerances.

Correctness. Sequence separation is >= 6 in both parse_contacts and candidate_sets, matching exp89's MIN_SEP = 6; long range is >= 24 in all five sites. Pairs are canonicalised via sorted(...) so (j,i) is never double-counted, and the flat i*L + j encoding decodes consistently. r_precision sorts descending with stable mergesort and cuts at labels.sum(), and is asserted equal to exp89's canonical scorer at runtime (analyze.py:219). sample_seed excludes model and temperature, so the two arms genuinely share seeds — re-verified sample-by-sample at analyze.py:260-268 and validate_results.py:52. Bootstraps resample proteins, not samples; paired contrasts use paired differences, and the only unpaired contrast is the disjoint high-vs-low Neff tier comparison. curveball preserves both margins (asserted at analyze.py:305-308, shuffle.py:39-40). thin_matched equalises n by construction. Headline numbers reproduce from the committed tables: diversity_contrasts.csv +0.10463 [0.08664, 0.12288] n=97; accuracy_summary.csv 0.55398 → 0.46283; curve_summary.csv 5641.9 → 8902.3 false discoveries (+57.8%).

Observation, not a finding (no rule covers it, non-blocking): summarize.py:290 reads data/rollout_diagnostics.csv.gz, which .gitignore excludes and restore_inputs.py does not restore, so a fresh clone can't re-run summarize.py without pulling that file from the HF root manually. The tables it derives are committed, and plots do regenerate from committed data/, so the closing-out requirement holds.

---

🤖 Code review

No issues found. Checked for bugs and AGENTS.md compliance.

Reviewed commit: e163932d1d93f73d1fab42795d797ba6d4693bd2

---

· branch codex/exp373-diversity-neff