[timodonnell]

The default exp277 checkpoint had eval-val results under the native side-chain-contact definition, which cannot be compared directly with historical CASP contact precision. This experiment rescores its archived 100-rollout predictions on all 97 natural eval-val monomers using Cβ <8 Å contacts (Cα for glycine), preserving the original metric as a control.

Long-range precision is 74.6% at L/5, 61.1% at L/2, and 45.9% at L. The report compares these descriptively with original CASP4/7/11–14 assessments and explains why a CASP rank or equivalent folding ability does not follow. Target composition matters: 68/97 proteins have recorded MSA depth ≥1000 and none have depth <10. Native-metric results for the previous exp232 default and the decontaminated-corpus KNN reference are included separately.

Includes frozen public eval-val inputs, per-protein results, confidence intervals, coordinate/tie audits, original timing provenance, a figure and a summary PDF. No new inference, eval-test scoring, or historical-target scoring.

Validation: all 1,940 archived metric rows and counts reproduce (maximum score difference 1.11e-16); three independent metric tests and Ruff pass. Anonymous download and rescoring reproduce the per-protein and aggregate files byte-for-byte. All 14 published artifacts were verified anonymously.

Related to #352. Public inputs and results: https://huggingface.co/buckets/open-athena/MarinFold/tree/data/exp352-casp-precision-context/v1

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-08T18:15:54.304196Z	527c212	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[claude]

Claude finished @timodonnell's task in 7m 18s —— View job

---

🤖 Code review

No issues found. Checked for bugs and AGENTS.md compliance.

Reviewed commit: 527c2124b29f06ccbaa58504fb9391fcf3c16097

---

• Gather context (root + experiments/AGENTS.md, PR diff, PR metadata)
• Run multi-agent correctness + compliance review (2 compliance agents, 2 bug agents, in parallel)
• Post findings

What was checked

AGENTS.md compliance — no violations found:

• Dir exp352_evals_casp_precision_context matches exp<N>_<kind>_<name>; frontmatter issue: 352, kind: evals, and title all match the dir and issue exp: eval-val contact precision and historical CASP context #352 title (experiments/AGENTS.md rules 1 & 4).
• No exception swallowing in any authored file — analyze.py, prepare_inputs.py, references.py fail loud via assert / raise. The one except ValueError in build_summary.py is verbatim scaffold-template code.
• No from __future__ imports, no in-function imports, no monkey-patching. analyze.py:18's sys.path.insert to reuse exp89's unmodified compute_metrics.py is path manipulation, not attribute replacement.
• Binaries: largest committed artifacts are plots/summary.pdf (171 KB), data/per_protein.csv.gz (118 KB), plots/precision_and_depth.png (88 KB). The 2.55 MB vote/coordinate bundle is correctly kept out of git on the public HF bucket and fetched anonymously via HfFileSystem(token=False) (rule 2).
• publish_to_hf.py targets hf://buckets/open-athena/MarinFold/data/exp352-casp-precision-context/v1 and its verify step genuinely re-reads remote bytes anonymously and SHA-256-compares.
• Summary-PDF requirements met: summary_narrative.md, committed plots/precision_and_depth.png.meta.json sidecar, plot_results.py uses save_plot_with_meta and reads only committed data/summary.csv.
• No W&B run, no checkpoint write, no training code → W&B routing, checkpoint-path, tokenizer, and history/ index rules don't trigger. Archived per-protein timings preserved in data/source_timings.csv with the canonical (stem, n_residues) schema.

Correctness — no bugs found in the areas most likely to break this kind of analysis:

• tie_statistics (analyze.py:29-46): threshold = np.sort(scores)[-top] guarantees 0 < remaining <= n_tied, so no division by zero or negative remaining; expected/min/max are the correct hypergeometric expectation and exact extremes.
• NaN handling: cb_resolved is derived from np.isfinite(coords).all(axis=1), so unresolved residues never enter the candidate universe — nan < 8.0 being False is belt-and-braces, not load-bearing.
• score += score.T cannot double-count: prepare_inputs.py:166 emits votes from the strict upper triangle (np.triu(score, 1)), diagonal stays zero.
• Cβ/Cα and indexing: label_seq - 1 gives 0-based indices matching exp89's universe, the deposited canonical sequence is hard-compared to the eval sequence before any index is used, CA is used only when canonical[index] == "G", and altloc selection is highest-occupancy-then-lowest-altloc.
• references.py pairing is genuinely paired — merge on (stem, range, cut) with validate="one_to_one" and both sides filtered to the same 97 stems.
• test_metrics.py's three tests were hand-traced and are real (not tautological, no relaxed tolerance); the boundary test correctly pins < 8.0 and confirms an unresolved high-vote residue stays out of the universe.
• Every headline figure in README.md and summary_narrative.md reconciles with the committed CSV/JSON (74.6/61.1/45.9; 81.2/70.7/57.5; 75.8/58.8/39.0; 38.8/54.8/84.3 at n=5/24/68; 2.32% random; median 84.1%; 82/97 ≥50%, 7/97 <20%; control_rows_checked=1940; maximum_control_error=1.11e-16; 97×100 = 9,700 rollouts; 8 audit stems).

PR-level — no problems. Related to #352 uses no issue-closing keyword (correct, since experiments/AGENTS.md rule 6 makes closing experiment issues human-only), and the agent-generated label is present.

One non-blocking observation, not flagged as an issue: README.md:66 cites the CASP7 source via a dspace.ceu.es PDF link while data/historical_context.csv cites the same Izarzugaza et al. work by DOI 10.1002/prot.21637 — a citation-style inconsistency, not a contradiction.
• branch exp352/casp-precision