Repository navigation
exp368: benchmark guidance and confirm rollout-weighted contact maps - #370
Draft
timodonnell wants to merge 49 commits into
Draft
timodonnell wants to merge 49 commits into
timodonnell wants to merge 49 commits into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The released proofreader can identify better next contacts, but its guided rollouts can become shorter, lose coverage, or exploit scoring errors. This experiment compares 15 guidance/search policies, threshold mixtures, fixed-length controls, and ordinary rollout aggregation using complete contact-map metrics and measured GPU costs.
All phases are complete: 1,920 guided trajectories on 32 pilot proteins; ordinary-pool validation on 1,375 structures on H100 and GB200; and one frozen reserved comparison on 1,473 structures in 295 related groups. No guided method or tested threshold mixture improves the measured ordinary-pool quality/cost frontier. The four-rollout 10/25/50% mixture loses 0.02587 R-precision against top-50% alone (95% paired interval −0.03373 to −0.01777).
The selected practical change is 128 ordinary rollouts with votes weighted by squared predicted rollout F1. On the reserved comparison, R-precision improves from 0.54501 to 0.55021: +0.00520 (95% paired group interval 0.00299–0.00767); long-range improves by 0.00419 (0.00199–0.00657). Mean warm GB200 cost is 5.477 versus 3.620 seconds per map. The plan was committed before new test generation; only these two configurations were evaluated, after every input completed. The cohort was previously used for the proofreader release, and generator pretraining decontamination is not established.
Includes a detailed report, 47-page PDF, 27 figures with generating commands, per-protein metrics/timings, immutable execution source bundles, and recovery/compute ledgers. Public artifacts: https://huggingface.co/buckets/open-athena/MarinFold/tree/data/exp368/reports. Report: https://github.com/Open-Athena/MarinFold/blob/codex/exp368-proofreader-guidance/experiments/exp368_evals_proofreader_guided_contact_maps/README.md.
Validation: 11 CPU correctness tests passed; real-model H100 cache and GB200 full-forward replay checks passed at their original tolerances. Failed BF16/GB200 cached-search paths were excluded. All 188,544 reserved trajectories retain identical raw/scored contact and token identities; all 48,030,172 reader contact probabilities are finite and in range; source/plan pins and paired-table identities were verified. All owned GPU jobs are terminal, and native bootstrap resources were removed. Warm compute totals exclude loading, staging and preempted work; operational incidents are disclosed in the execution appendix.
Stacked on #358. Experiment tracking: #368. No experiment issue was closed and no shared cluster configuration was changed.