Skip to content

exp368: benchmark guidance and confirm rollout-weighted contact maps - #370

Draft
timodonnell wants to merge 49 commits into
codex/exp356-proofreaderfrom
codex/exp368-proofreader-guidance
Draft

timodonnell wants to merge 49 commits into
codex/exp356-proofreaderfrom
codex/exp368-proofreader-guidance

Conversation

@timodonnell

@timodonnell timodonnell commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

The released proofreader can identify better next contacts, but its guided rollouts can become shorter, lose coverage, or exploit scoring errors. This experiment compares 15 guidance/search policies, threshold mixtures, fixed-length controls, and ordinary rollout aggregation using complete contact-map metrics and measured GPU costs.

All phases are complete: 1,920 guided trajectories on 32 pilot proteins; ordinary-pool validation on 1,375 structures on H100 and GB200; and one frozen reserved comparison on 1,473 structures in 295 related groups. No guided method or tested threshold mixture improves the measured ordinary-pool quality/cost frontier. The four-rollout 10/25/50% mixture loses 0.02587 R-precision against top-50% alone (95% paired interval −0.03373 to −0.01777).

The selected practical change is 128 ordinary rollouts with votes weighted by squared predicted rollout F1. On the reserved comparison, R-precision improves from 0.54501 to 0.55021: +0.00520 (95% paired group interval 0.00299–0.00767); long-range improves by 0.00419 (0.00199–0.00657). Mean warm GB200 cost is 5.477 versus 3.620 seconds per map. The plan was committed before new test generation; only these two configurations were evaluated, after every input completed. The cohort was previously used for the proofreader release, and generator pretraining decontamination is not established.

Includes a detailed report, 47-page PDF, 27 figures with generating commands, per-protein metrics/timings, immutable execution source bundles, and recovery/compute ledgers. Public artifacts: https://huggingface.co/buckets/open-athena/MarinFold/tree/data/exp368/reports. Report: https://github.com/Open-Athena/MarinFold/blob/codex/exp368-proofreader-guidance/experiments/exp368_evals_proofreader_guided_contact_maps/README.md.

Validation: 11 CPU correctness tests passed; real-model H100 cache and GB200 full-forward replay checks passed at their original tolerances. Failed BF16/GB200 cached-search paths were excluded. All 188,544 reserved trajectories retain identical raw/scored contact and token identities; all 48,030,172 reader contact probabilities are finite and in range; source/plan pins and paired-table identities were verified. All owned GPU jobs are terminal, and native bootstrap resources were removed. Warm compute totals exclude loading, staging and preempted work; operational incidents are disclosed in the execution appendix.

Stacked on #358. Experiment tracking: #368. No experiment issue was closed and no shared cluster configuration was changed.

@timodonnell timodonnell added the agent-generated Opened by an AI agent label Oct 21, 2026
@timodonnell timodonnell changed the title exp368: evaluate proofreader-guided contact-map inference exp368: benchmark guidance and confirm rollout-weighted contact maps Oct 22, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-generated Opened by an AI agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant