[timodonnell]

Measure contact-conditioning and protein-length effects for the default exp277 step 266344 checkpoint on all 97 eval-val proteins, and compare MarinFold with original AFDB training structures and later biological monomers. Includes source CSVs, figures, a summary PDF, frozen execution bundles, raw predictions, and public checksummed archives.

• Eight random true contacts or a 5% budget improve remaining-contact R-precision by about six percentage points. Mean accuracy peaks at 0.711 at 40%; no tested dose reaches 0.80. Secondary-structure coverage gives no clear early advantage.
• R-precision has no detectable monotonic relationship with protein length (Spearman rho -0.068, p=0.507).
• The corrected AFDB screen requires single-protein biological assemblies and an experimental-context audit. Of 20 source-coordinate-verified candidates, none is a clear catastrophic fold failure. Three moderate monomer comparisons use Pru p 7, RecJ, and MrpR/YopR. MarinFold favors AFDB for RecJ and MrpR; Pru p 7 has low accuracy and conformer-dependent preference across its 20-model NMR ensemble. Previous complex/state examples are explicitly superseded and excluded from the summary figures.

All 6,664 eval-val contact-injection conditions / 666,400 rollouts and 600 corrected monomer rollouts are complete. Supplied contacts are removed from truth and prediction candidates, with matched unconditioned comparisons and protein-bootstrap intervals. Monomer selection checks exact retained training membership, original AFDB v4 contacts, later PDB release, coverage, biological assembly, and literature context. Crystal copies are distinguished from multichain biological assemblies. The monomer screen is limited and does not establish error-correction prevalence or behavior on severe AF2 fold failures.

Validation: 12 protocol/assembly-regression tests pass; 20 existing saved-vote comparisons exactly match the canonical evaluator. No token-cap hits occurred. Invalid-rollout sensitivity preserves the monomer conclusions. All six new inference markers match the frozen plan, worker, checkpoint identity, and H100 execution bundle. Anonymous access and all 313 members of the updated 16.97 MB public result archive were verified; the 97 eval-val raw-prediction bundles remain separately indexed and checksummed.

References #369.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-21T17:21:15.199297Z	8042714	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[claude]

Claude finished @timodonnell's task in 10m 7s —— View job

---

Review complete

• Gather context (root + experiments/AGENTS.md, PR metadata, experiment README)
• Run 4 parallel reviewers (2× AGENTS.md compliance, 2× bug hunt)
• Validate candidate findings — none survived to validation (all four returned empty)
• Post review record

---

🤖 Code review

No issues found. Checked for bugs and AGENTS.md compliance.

Reviewed commit: 804271405587b92deecfcd806de272bbc1a1132c

---

What was checked

Four independent Opus reviewers (two on AGENTS.md / experiments/AGENTS.md compliance, two on correctness) read all 22 Python files plus the committed data/ and plots/ artifacts. Every candidate finding was cleared by the reviewer that raised it; nothing reached the validation stage.

Compliance — clean on the rules in scope:

• Error handling. Only two except blocks in the whole directory. screen_training_errors.py:105-120 catches per-candidate fetch/parse errors into a ledger and then raise RuntimeError(...) if any occurred — catch-with-context then fail-loud. build_summary.py:247-250 is byte-identical to scripts/templates/build_summary.py. The dispatch/worker/publish scripts contain zero except blocks and raise on bad provenance, checksum mismatch, missing completions, and unexpected finish_reason.
• Imports. No from __future__. The three in-function import sites (dispatch_spiking.py s3fs/fray/iris, spiking_worker.py torch/vllm/transformers, build_summary.py matplotlib) are optional-dependency guards — fray/iris/torch/vllm are genuinely absent from the experiment's pyproject.toml/uv.lock and only exist in the submission path or the foreign container.
• No backward compat / no monkey-patching. No hasattr probes, fallback paths, deprecation shims, or runtime attribute replacement.
• Checkpoint path. exp277-step-266344 carries both run identity and step; no final//latest/. stage_model.py verifies size + SHA-256 against exp277's publication_manifest.json before loading.
• Storage. Large output goes to the sanctioned CoreWeave bucket s3://marin-us-east-02a/MarinFold/exp369/, co-located with the cw-rno2a jobs; public artifacts to hf://buckets/open-athena/MarinFold/, read back with HfFileSystem(token=False) so reproduction needs no auth. collect_run.py:34-45 really does gate — the ≤10 GB budget is compared against the inventoried total before any download starts.
• Layout / artifacts. kind: evals, frontmatter issue: 369 matches the dir name. plots/summary.pdf, summary_narrative.md, and a .meta.json sidecar for all 6 PNGs are present; build_summary.py only reads saved PNGs + narrative, no analysis at PDF-build time. data/timings.csv carries a superset of the root-AGENTS.md minimum schema, so it joins on (stem, n_residues) with exp12/exp20. No weights/parquets/prediction dumps in git. No wandb usage, so no history/runs/ obligation.

Correctness — the claims that most needed verifying hold up in the code:

• No leakage. protocol.py:106-117 applies keep &= ~excluded[ii, jj] to the candidate index arrays before labels = positives[ii, jj], so supplied pairs leave the candidate set and the truth count together, and n_true is recomputed per range. n_true == 0 returns r_precision: None, not 1.0.
• Matched comparator. analyze_spiking.py:70-73 calls the conditioned, unconditioned, and both validity-gated variants with the same supplied/separation, so the dashed baseline is genuinely rescored on the identical remaining universe.
• Ranking matches exp89. np.argsort(-scores, kind="stable")[:n_true] over triu_indices(length, k=min_separation) reproduces the canonical evaluator's mergesort order, which is why validate_metrics.py:42 can assert agreement at rtol=0, atol=0.
• Nestedness, pair canonicalization, bootstrap axis. Every dose is a prefix of one fixed per-(policy, repeat) order. spiking_worker.py re-canonicalizes to sorted sequence-space pairs after the mod-2000 ring unwrap. Bootstraps resample the per-protein vector (proteins, not conditions), and paired comparisons bootstrap already-differenced per-protein values.
• Sharding. index % n_shards over a deterministically sorted list is disjoint and exhaustive; rebalance_tail.py:57 asserts the redistributed partition is a permutation of the unfinished counts, and parts <= len(counts) prevents empty groups. The zero-dose special case (["random"] only when k == 0) is applied identically in all six places that enumerate conditions.
• Gates and publishing. publish_predictions.py uploads with check=True before writing the index that names the object; publish_to_hf.py hashes the final archive bytes and excludes public_artifacts.json from its own archive, keeping content-addressing stable.
• Misc. end_id is looked up via convert_tokens_to_ids("<end>") rather than hardcoded; Kabsch in plot_structures.py handles the reflection determinant correctly; spearmanr (not Pearson) in plot_length.py; distances are Å and Cα throughout; exclusive_preference implements ties-count-half size-balanced and matches the 0.75/0.25 expectations in test_protocol.py:91-92; length_bins.csv uses right=False with an np.inf top edge so neither extreme is dropped, and the bin counts sum to 97.

Caveats

• I could not execute test_protocol.py — running uv run pytest requires a permission grant this job doesn't have, so the eight protocol tests were read, not run. The PR states they pass. The assertions are substantive invariants (a supplied pair cannot score itself, retractions don't vote, out-of-range pairs are counted without discarding valid ones, SSE coverage precedes coil-only contacts), not tautologies, and none relax a tolerance.
• PR description: no issue-closing keyword (it correctly says "References exp: default-model contact spiking, length dependence, and AFDB error preference #369", so merging won't close the experiment issue), and the agent-generated label is present. Nothing to flag.
• Two input-dependent sharp edges were considered and deliberately not flagged, since neither can fire on the committed data: summarize_contexts.py:34 divides by n_coverable_elements without a zero guard, and summarize_results.py:51 calls interval() unguarded for empty groups (the analogous call at analyze_spiking.py:107 is guarded). Worth knowing if either script is ever pointed at a different cohort.

--- · Branch exp369/default-model-analyses

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8042714055

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Pin the Marin launcher dependency

When the submission or recovery scripts are run from a clean checkout with uv run, their fray and iris imports fail because this dependency list and the committed lock contain none of the Marin launcher stack. The README notes that the completed run relied on a separate task-local checkout at Marin revision af9c6789e8, so the committed environment cannot reproduce or retry the documented launch; pin that revision and include it in uv.lock.

AGENTS.md reference: AGENTS.md:L358-L363

Useful? React with 👍 / 👎.