Repository navigation
exp354: short rollouts reduce precision but improve AUC on eval-val - #355
timodonnell wants to merge 6 commits into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Claude is reviewing this PR…
|
timodonnell
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d71d89d138
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| (bundle / "targets.jsonl").write_text("".join(json.dumps(r) + "\n" for r in selected)) | ||
| for name in ("worker.py", "contacts.py"): | ||
| shutil.copy2(HERE / name, bundle / name) | ||
| shutil.copy2(ROOT / "experiments/exp89_evals_contacts_v1_model_on_eval_set/compute_metrics.py", bundle) |
There was a problem hiding this comment.
When this experiment is submitted, it copies compute_metrics.py directly from exp89, making exp354 another consumer of stable experiment-local metric code and coupling reruns to another experiment's implementation. Promote these shared metric functions to the eval kind library and import or bundle them from there, as required once a helper has a second use case.
AGENTS.md reference: experiments/AGENTS.md:L256-L263
Useful? React with 👍 / 👎.
On the current exp277 default (step 266344), none of the five 1,000-short-rollout settings beats 100 full rollouts on eval-val contact precision. The best short cap, L/2, scores 0.55003 all-range R versus 0.55743 for full rollouts: paired delta −0.00740 [−0.01094, −0.00396]. Its AUC improves from 0.93921 to 0.95565, so this is a precision-versus-global-ranking tradeoff.
Adds the reproducible contact-count sampler, nested 100-short controls, paired bootstrap analysis, per-input timings, raw-sample audit, report, and summary PDF. Only the 97 eval-val proteins were scored. The shared generation through max(20, L/2) took 25.0 minutes versus 6.9 minutes for the full baseline on one GB200; smaller-cap standalone runtimes were not measured.
Validation:
Report · Public artifacts
Refs #354, closed as completed at the user's explicit request after the final report and summary PDF were posted.