exp368: benchmark guidance and confirm rollout-weighted contact maps - #370

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

PR-level rules

• No issue-closing keyword targeting an experiment-labelled issue. #368 / #358 are plain references; "No experiment issue was closed" is prose with no issue number attached (experiments/AGENTS.md rule 6).
• codex/* branch carries the agent-generated label (rule 5).
• Frontmatter accurate: issue: 368 matches exp368_, kind: evals matches the dir segment, branch: matches the head branch.

AGENTS.md compliance — code

• Exceptions. Exactly one except in the whole directory: build_summary.py:267. I diffed it against scripts/templates/build_summary.py — that block is unchanged scaffold template code, so it's pre-existing and out of scope. Every other error path raises; the code is notably fail-loud (engine.py:76 nonfinite scores, score_pool.py:53 on incomplete source, analyze.py:93 hard-fails on an incomplete reserved cohort before reading labels, backend_check.py:128 raises after writing results so a failed replay can't pass silently).
• No from __future__, no hasattr-shims, no setattr, no monkey-patching. prepare_regional_mirror.py:13 subclasses s3fs.S3FileSystem — sanctioned option 2 in the root "Never monkey-patch" escalation list.
• Row dropping. Every continue/break is either resume logic gated on a durable .done.json, or a recorded skip — compare.py:36 writes .skipped.json and prints the count; summarize_compute.py:29 refuses any phase whose completeness.csv isn't complete.
• Tests tighten, not relax. test_metrics.py:16 uses rtol=1e-12 against sklearn's average_precision_score as an independent oracle; test_engine.py:41 and test_length_probe.py:22 assert bitwise (rtol=0, atol=0) cache isolation; test_analysis.py:45 is a negative test asserting non-zero exit.

AGENTS.md compliance — artifacts & infrastructure

• No binaries in git. Largest file is the required plots/summary.pdf (2.3 MB); everything else is text. No *.pt/*.safetensors/*.parquet/*.npy. The ~400 MB of rollout parquets stay on S3 with only hash manifests committed.
• Public artifacts. publish_to_hf.py:18 targets hf://buckets/open-athena/MarinFold/data/exp368/reports and re-verifies readback via HfFileSystem(token=False), proving anonymous readability.
• Region co-location. Zero gs:// references — all storage is CoreWeave S3 under one MarinFold/exp368/ prefix. launch.py:44 rejects cluster/region mismatches; prepare_regional_mirror.py:30 and regional_inputs.py:18 assert x-amz-bucket-region before any read. Cross-region transfer is 6.01 GB, under the 10 GB sign-off threshold, and prepare_regional_mirror.py:43 refuses ≥ 7 GB with a durable reservation re-checked per chunk.
• W&B / run history: N/A — nothing calls wandb.init() (the only wandb hit is an rsync exclude), so neither the routing rule nor history/runs/*.md applies.
• Timings. All 28 timings CSVs carry every one of the 16 columns in the root "Capture timings" schema, plus allowed per-predictor extras.
• Summary PDF. 47 content pages (20 narrative + 27 plot slides); all 27 PNGs have committed .meta.json sidecars with real script + full args; build_summary.py reads only saved plots/CSVs, reruns no analysis.

Bug hunt — metrics & statistics

• R-precision uses L = len(native) (true contacts) and sorts descending with a label-independent SHA-256 tiebreak; precision_L/L2/L5 separately use residue count, intentionally distinct.
• Eligible-pair universe (L-s)(L-s+1)//2 is exactly sum_{d=s}^{L-1}(L-d). Separation filter is consistently b - a >= separation across aggregate, map_metrics, analyze_reranking, and the released parser — upper-triangle only, no diagonal, no (j,i) double count.
• Residue indexing is 0-based end-to-end and round-trips: make_prompt emits <p{(offset+i)%2000}>, parse_contacts inverts it, map_metrics bounds-checks.
• Paired intervals keep the pairing: both arms set_index('entry_id').sort_index() before subtracting, compare.py:42 asserts group_id identity, :27 rejects duplicate keys so no join changes the reported N. bootstrap resamples groups via a single shared index array into both total and count — matching the "paired group interval" claim. NaN drops symmetrically because delta is NaN iff either arm is.
• Squared-F1 weights are strictly positive and finite for every power in {1,2,4,8}; non-renormalization is harmless since all downstream use is rank-based.
• Headline numbers reproduce from the committed CSVs. data/reserved_test/paired.csv → 0.5450070 → 0.5502065, Δ 0.0051995, interval (0.0029947, 0.0076662); long-range 0.0041860 (0.0019921, 0.0065682) at 1473 structures / 295 groups. data/pilot_mixtures/paired_vs_top50_4.csv → -0.0258747 (-0.0337301, -0.0177729). Both match the PR body.

Bug hunt — engine & execution

• Contact-end readout position is prompt_length + 3i + 2 in all three places that compute it (worker.py:66, worker.py:87, test_engine.py:45), correctly excluding the trailing <end>.
• All special-token ids come from tokenizer.convert_tokens_to_ids; no assumed literals outside the synthetic test tokenizer.
• Top-p order is the standard one: grammar mask → softmax → cumsum → exclusive-prefix removal (boundary token kept) → re-softmax. All search/selection sorts maximize (descending=True, reverse=True, max).
• Cached-search path is pinned against a full forward to 1e-6 (test_engine.py:82) and forced to FP32 (engine.py:107), consistent with the disclosed BF16/GB200 exclusions.
• Sharding interleaves (work[shard::shards] over a cost-descending sort) rather than contiguous blocks — exactly what the root AGENTS.md fan-out section asks for. No item dropped or double-assigned.
• Timing is CUDA-synchronized on both sides of inference, with model_load_seconds measured separately; per-rollout seconds are summed per-map downstream, so the seconds-per-map figure isn't a per-rollout/per-shard mix-up.
• One generator per rollout seeded from stable_seed('sample:exp368:<entry>:r<sample>'), so the 128 rollouts are genuinely independent streams — no key reuse collapsing the effective pool.
• serve.py:134 polls children independently, so one failure can't abandon the remaining waits (the job.wait() trap in AGENTS.md). pool_bootstrap.sh:25 takes an ephemeral kernel-assigned VLLM_PORT rather than a fixed default or $$/$(hostname).

Deliberately not flagged (input/state-dependent or explicitly in-scope-for-experiments)

• engine.py:161 (arange(length) + offset) % 2000 would collide if offset + length > 2000; max L in the frozen plans is 937, and worker.py:169 asserts cached pairs equal the released parser's, which would raise on collision.
• analyze_mixtures.py:64 includes hardware in the groupby keys, so heterogeneous-GPU rotations would yield >1 row per protein; didn't happen in the committed run, and compare.py:27 would have raised.
• Three slightly different "predicted precision" definitions across aggregate, observations, and engine.utility — each a defensible trajectory-quality scalar, none making a metric wrong.
• serve.py:87 mirror_committed omits *.verification.json; diagnostic-only, .candidates.parquet is covered.
• Hard-coded paths / run names / magic constants inside the experiment dir, and uv.lock churn — normal per experiments/AGENTS.md.

Not verified

I could not execute the test suite in this sandbox (uv run pytest needs approval here), so the PR's "11 CPU correctness tests passed" claim is taken from the body rather than re-run. Per the skill, linters and type-checkers were not run.

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".