[timodonnell]

Adds a reproducible contact-map audit of all 97 eval-val proteins using exp277 epoch 2, step 479417. The original offline atlas shows predicted and true maps, votes, errors and annotated cases. The follow-up saves 200 complete individual rollouts per protein and adds an offline viewer for all 19,400 maps, with sample selection, paired truth, synchronized zoom, error overlays, per-sample accuracy and CSV downloads.

The whole-map probe finds that selecting one sample does not improve mean accuracy even with a truth-informed oracle: pooled top-R F1 is 55.79%, versus 55.04% for the best of 200 (paired Δ −0.75 points, 95% interval −2.00 to +0.57). Pooled voting beats the best individual on 62/97 proteins. Average individual F1 is 41.53%; independent-pool consensus and mean-token-likelihood selectors score 48.95% and 46.73%. Severe failures remain poor in their individual maps, although 35 proteins have a better oracle sample. The report exposes those exceptions.

Independent-pool cluster voting is tied with pooled voting, but average-linkage Jaccard clustering mostly isolates outliers: only 37/194 k=4 fits have multiple eligible test clusters. This limits the modes interpretation. The writeup distinguishes observed map errors from unproven training causes and 3D feasibility.

Validation: all 19,400 rollouts terminated and passed raw-data integrity checks. Both independent 100-rollout pools reproduce saved all/long scores within 0.005. There are 582 exact comparisons against the canonical scorer, in addition to the original 1,940 source-score checks. Browser checks match every individual map in both ranges (38,800 scores) at desktop and mobile sizes and exercise every gallery map and control. Independent fixtures cover cardinality, emission order, a two-mode mixture and truth-independent selection. Ruff and diff checks pass. No eval-test data were read.

The run records per-protein timings, checkpoint/tokenizer identities, sampling seeds, source hashes and all execution attempts. Pure inference took 19.15 minutes on one H100. The in-region fp32→bf16 export verifies all 267 tensors and preserves the weights used by bf16 inference. Reports, raw maps, tables and figures are anonymously readable and checksum-verified; a clean-cache public download reproduced all samples.

Download the whole-map HTML report · All public artifacts. Small analysis artifacts and plots remain in git; the 17.5 MB HTML and 11.9 MB raw archive are published to the bucket.

Refs #345. Leave the experiment issue open for human follow-up.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-30T21:50:05.744950Z	1ca9a34	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[claude]

Claude finished @timodonnell's task in 9m 38s —— View job

---

/review-pr --comment 346 — complete

• Preconditions: PR open, non-draft, agent-generated label present, no prior review of 1ca9a34
• Guidance files in scope: root AGENTS.md, experiments/AGENTS.md
• PR-level rules (issue-closing keywords, agent label)
• 2× AGENTS.md compliance agents (opus, parallel)
• 2× bug agents (opus, parallel) — Python analysis + report JS/HTML
• Posted review record

Result: no issues found. Review comment: #issuecomment-5920387507

PR-level checks

Refs #345 is not an issue-closing keyword, so merging won't close the experiment issue (experiments/AGENTS.md rule 6). The agent-generated label is present. No description problems.

Compliance

Frontmatter matches (issue: 345, kind: evals ↔ the _evals_ dir segment, title byte-identical to the live issue title). branch: main is correct usage — per AGENTS.md#L128-L131 that field records where an experiment lives, not the short-lived PR branch, and six merged siblings use it the same way.

plots/summary.pdf present with *.meta.json sidecars on all three PNGs; build_summary.py is the scaffold template verbatim modulo one import relocation and reads only committed inputs. No swallowed exceptions, no from __future__, no in-function imports, no monkey-patching, no W&B usage, nothing written to GCS. Checkpoint identifiers carry both run name and step (…epoch2-from213072-1.5B + step479417).

Two things I checked and deliberately did not flag:

• data/report_inputs.json.gz is gitignored, and plot_results.py / build_report.py read it unconditionally. But analyze.py:201-202 auto-downloads it anonymously via HfFileSystem(token=False), and the "Closing out" rule constrains rerunning the pipeline, not network access. A 2.1 MB bundle on the public bucket is exactly the data/-vs-HF split rule 2 prescribes. publish_to_hf.py round-trips to the same path analyze.py fetches.
• No data/timings.csv — the timings rule is scoped to "whenever you run a predictor," and this PR runs none; it re-analyzes saved exp277 matrices.

Correctness

The metric path holds up. Spot-verified against experiments/exp89_.../compute_metrics.py:

• JS/Python top-R agree exactly. Python's np.argsort(-s, kind="mergesort")[:R] over np.triu_indices-ordered candidates gives (score desc, i asc, j asc); the JS comparator b[2]-a[2]||a[0]-b[0]||a[1]-b[1] (report_template.html:43) is that same total order. This matters — cutoff_tie_size reaches 188, so a mismatched tiebreak would be visible.
• The JS ranks only vote-bearing pairs while Python ranks all candidates. Safe here, and deliberately guarded: build_report.py:24 asserts supported >= L, and min cutoff_votes is 8, so the top-R/L/L-5 prefixes are entirely within ranked. check_report.py then pins all 97 selections to r_precision at 1e-12.
• Range boundaries match canonical RANGES (short 6–11, medium 12–23, long ≥24, all ≥6); L/5 floors identically (max(1, L//5) ↔ Math.max(1, Math.floor(L/5))).
• truth &= candidate at analyze.py:130 before metric_rows is a no-op for the scorer — it only reads tmat[i, j], exactly the set candidate marks.
• The one-to-one oracle is a genuine maximum bipartite matching; np.sum(perm >= 0) counts it correctly under either perm_type. distance_transform_cdt(~truth) has the right polarity, and maximum_filter(size=2*t+1) is the intended ±t-per-endpoint dilation.
• test_analysis.py fixtures hand-compute correctly and do isolate what they claim: the ±19-separated FP genuinely cannot rescue the already-recovered contact.
• allow_nan=False is not a hazard — data/per_protein.csv (the only table embedded) has zero empty or nan cells; the NaN-producing rows live in range_diagnostics.csv / canonical_metrics.csv, which are not embedded.

Caveat on method

Code execution was denied in this sandbox, so all of the above is static analysis plus cross-checks against the committed CSVs and provenance.json — I could not run analyze.py, test_analysis.py, or check_report.py, nor decode the report payload. The PR's own claims about browser behavior and the 1e-12 source reproduction rest on the author's run, not on a re-execution here. Two minor cosmetic items surfaced and were not flagged: the unresolved-residue gray stripes are drawn a half-cell off from the contact dots they annotate, and a deeply nested manual zoom could in principle collapse x1 === x0.
· exp345/contact-map-report

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1ca9a34868

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Move the shared scorer into an evals library

This makes exp345 another direct consumer of exp89's metric helpers by mutating sys.path; repo-wide search shows the same helpers are already reused by exp321, exp326, exp333, and others. Once a second experiment needs a helper, repository policy requires promoting it to the kind library, so the canonical scoring API should live in an evals package imported through an explicit path dependency rather than treating exp89 as a library.

AGENTS.md reference: AGENTS.md:L38-L42

Useful? React with 👍 / 👎.

[claude]

🤖 Code review

No issues found. Checked for bugs and AGENTS.md compliance.

Reviewed commit: 1ca9a34868e5072a0df1035bd23fcb48040a65c1