Repository navigation
exp350: evaluate pair-held-out FoldBench complexes - #351
timodonnell wants to merge 16 commits into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8f396ad493
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| The one-time complex decoder and Helico reference builders are | ||
| `build_complex_sequences.py` and `prepare_helico_reference.py`. Search commands, | ||
| hashes, database counts and source provenance are committed under `data/`; large | ||
| alignment databases and logs remain under `/data/exp350`. |
There was a problem hiding this comment.
The full alignment TSVs and logs that underpin the exclusion and no-hit claims are left only under local /data/exp350, so external reviewers cannot inspect them and a local disk loss would permanently remove the evidence, especially important given the documented prefilter-cap caveat. Consolidate the relevant outputs in the public open-athena/MarinFold HF bucket, link them here, and include the required reproducible publisher.
AGENTS.md reference: experiments/AGENTS.md:L61-L76
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 339c115. The complete six broad alignment TSVs, logs, manifests, and both higher-cap pair-confirmation TSVs/logs are now published in the public MarinFold HF bucket; data/evidence_manifest.json records sizes and SHA-256 hashes.
| query_sha256 = hashlib.file_digest(args.queries.open("rb"), "sha256").hexdigest() | ||
| run_tag = f"v3_{query_sha256[:12]}" | ||
| query = args.work / f"{args.arm}_queryDB_{run_tag}" | ||
| run([MMSEQS, "createdb", args.queries, query, "--shuffle", "0"]) |
There was a problem hiding this comment.
When the documented search is rerun with an unchanged query FASTA and the default /data/exp350 workdir, the content-derived tag selects the same query, result, and temporary database paths, and this unconditional createdb encounters the prior run's files. MMseqs does not overwrite these databases; the established wrapper in experiments/exp213_evals_train_sequence_overlap_audit/search_overlap.py:168-174 explicitly removes query/result/tmp artifacts for this reason. Clear those hash-tagged artifacts before starting, reuse a verified complete run, or create a unique work path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 339c115. Every run now clears the hash-tagged query/result/tmp database prefixes and the final TSV before rebuilding, so a same-query rerun cannot silently reuse stale MMseqs outputs.
| "source_per_complex_sha256": hashlib.file_digest( | ||
| (args.data / "per_complex.csv").open("rb"), "sha256" | ||
| ).hexdigest(), |
There was a problem hiding this comment.
In the documented workflow, this hashes the intermediate per_complex.csv, but the later final analyze_survival.py invocation rewrites that file with native/Helico statuses. The committed provenance already records 68202a21… while the committed CSV hashes to d2da618f…, so the source hash cannot verify any committed artifact even though the selected FASTA is consistent. Hash immutable selection inputs/columns or regenerate the provenance after the final analysis.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 339c115. Reduced-query provenance now hashes only immutable selection columns plus query membership and the source query FASTA; later status columns no longer invalidate the recorded selection.
timodonnell
changed the title
timodonnell
changed the title
Adds a reproducible FoldBench protein-complex evaluation for the exp343 multichain checkpoint: pair-level homology auditing, a frozen complex cohort, contact R-precision, Helico structural evaluation, sampling controls, and visual failure diagnostics. The four-model comparison measures both intra-chain and inter-chain accuracy from predictions of the same complete dimers.
The audited source benchmark contains 30 natural dimers. The frozen context-complete contact subset has 23 targets (6 development, 17 test); both-chain training-pair matches are excluded under the 30% identity rule. Individual component-family novelty is not claimed. Seven context-ineligible targets remain explicitly identified.
Test mean R-precision (identical 17 complexes and experimental resolved-residue masks):
Both MarinFold arms use 100 rollouts. The default arm removes all linker positions before scoring. Intra-chain R-precision pools eligible pairs within the original two partners; per-partner and long-range scores are also included. Structure models select one of five samples using their own confidence, then rank contacts by pyconfind degree. AF3 receives paired/unpaired MSAs and no templates; external-baseline training-set decontamination is not established here. This is an exploratory follow-up on the previously read test split.
Helico all-contact top-L, selected on development, reaches test mean DockQ 0.0697 versus 0.0546 with intra-chain contacts, 0.0362 with contacts withheld, and 0.8010 with oracle contacts. Only 2/17 predicted-contact structures reach DockQ 0.23.
The 1,000-rollout follow-up preserves all targets and gives unfinished attempts zero credit. Oracle best@1000 F1 is 0.1923, versus 0.0729 for amino-acid-pair-matched random and 0.1348 for residue-degree-preserving random. The degree-control advantage is +0.0575 [0.0389, 0.0878]; it supports pairing information beyond residue localization, but no sample reaches F1 0.5. Oracle selection requires experimental truth. Consensus inter-chain R-precision remains only 0.0373 at 1,000 rollouts.
The contact atlas distinguishes wrong contacts from absent contacts. A separate five-page four-model PDF contains aggregate accuracy, per-complex heatmaps, intra-versus-inter scatter plots, and per-partner accuracy. Small plotting inputs and source/timing manifests are committed; structures and raw rollouts are public.
Validation: 26 tests pass; Ruff and whitespace checks pass. Native inter-chain truth is reproduced exactly for all 23 targets. Every predictor covers all 23 targets; 184 predictor timing records are saved. Both 100-rollout arms complete 2,300/2,300 attempts. Baseline input sequences and confidence-selected structures are verified. ESMFold2's original caller hit a metadata serialization error after saving each result; all outputs were recovered and validated without rerunning or discarding predictions, and the wrapper is fixed.
Tracks #350. Helico runner/scoring fixes are in Open-Athena/helico#19.