Repository navigation
exp: fine-tune Qwen3.5 base models for protein contacts #347
Description
Question
Can pretrained Qwen3.5 base language models learn protein contacts more effectively than our models trained from scratch? Does a natural-language instruction and ordinary protein sequence improve learning relative to directly continuing pretraining on contacts-v1 documents?
Hypothesis
General pretraining may supply useful sequence, indexing, and output-format priors. A plain-language contact-prediction prompt may make those priors easier to use. The effect may vary with model size.
Approach
- Fine-tune the pretrained base checkpoints Qwen/Qwen3.5-0.8B-Base, Qwen/Qwen3.5-2B-Base, and Qwen/Qwen3.5-4B-Base. Pin revisions and retain their pretrained tokenizers.
- Compare two document formats at each size: existing contacts-v1 documents; and a simple natural-language format description followed by the protein sequence and the corresponding contacts.
- Use matched examples from the existing exp225/exp232 decontaminated corpus and document all transformations, tokenization expansion, masking, and truncation decisions. Use text-only model components; train the language model weights, not a randomly initialized replacement.
- User-approved first-pass budget: approximately 1B training tokens per trial, six trials, at most 48 H100 GPUs concurrently. Begin with correctness and GPU smoke tests, then batch-priority CoreWeave training. No bulk cross-region data transfers.
- Log to open-athena/MarinFold, record run history, retain resumable checkpoints with tokenizer and pinned configuration, and compare validation contact likelihood, output validity, and contact-prediction metrics. Use eval-val for development; do not tune on eval-test.
Success criteria
All six training runs complete their declared budget and have recoverable model/tokenizer artifacts. Report data and compute exposure, learning curves, and matched contact-prediction validation results, with honest limits on attribution to pretraining and document format. This initial experiment has no matched random-initialization control, so comparisons against existing models are contextual rather than a causal pretraining ablation.
Metadata
- Kind: models
- Requested by the user: file the experiment and run it.
Activity
timodonnell commentedon Oct 2, 2026
🤖 Implemented the experiment in draft PR #348 and launched the first four production trials: 0.8B and 2B × contacts-v1 and prompted documents, each targeting approximately 1B tokens, with eight H100s per trial. The six-trial plan remains within the user-approved 48-H100 limit.
The paired corpus is complete: 920,611 training and 9,235 validation proteins, with complete sequences/contact lists in both formats. Their token totals are 4.493B versus 1.290B, so equal token budgets imply different protein exposure. The revisions, source hashes, filtering counts, validation timings, and intermediate learning curves are committed in the experiment directory.
GPU training/checkpoint checks passed for 0.8B and both 2B formats. The 0.8B recovery check restored optimizer/RNG/data state and exactly reproduced checkpoint validation NLL before continuing. Eleven experiment tests, eight capacity-reader tests, Ruff, and Pyrefly passed. The first 0.8B contacts-v1 run improved contact-token validation NLL from 0.70685 to 0.59605 at 36.92M tokens; this is likelihood evidence, not contact accuracy.
Current execution constraint: Kueue preempted the three active production jobs. All four production roots remain submitted for automatic retry; the first has a step-253 checkpoint. The other two preempted runs had not saved their first periodic checkpoint. Exact task diagnostics show scheduling gates, and read-only inspection found the apparently idle whole-GPU nodes are cordoned. No cluster configuration or priority was changed. A separate eight-H100 saved-model generation canary is also queued, bringing the submitted total to 40 GPUs. Its generation path has not yet completed GPU validation.
The 4B trials are pending explicit model-transfer approval under the repository's >10GB cross-region-copy rule: all three pinned base models total approximately 15.6GB. Only 0.8B and 2B (approximately 6.3GB) have been staged so far. No full-budget trial or final contact-accuracy comparison is complete.
Registered runs: 0.8B contacts-v1, 0.8B prompted, 2B prompted. The 2B contacts-v1 job is submitted but has not reached W&B initialization.
timodonnell commentedon Oct 2, 2026
🤖 The approved 4B transfer is complete.
/timodonnell/exp347-stage-4b-a01staged 9,342,808,116 bytes at the pinned revision; all three base-model tokenizers match exactly. File checksums are committed underdata/model_staging_4b.json.All six 1B-token trials are submitted on eight H100s each, at batch priority and within the 48-H100 cap. Current Iris roots are under
/timodonnell/:exp347-qwen35-0p8b-contacts_v1-1bt-a03exp347-qwen35-0p8b-prompted-1bt-a04exp347-qwen35-2b-contacts_v1-1bt-a03exp347-qwen35-2b-prompted-1bt-a03exp347-qwen35-4b-contacts_v1-1bt-a04exp347-qwen35-4b-prompted-1bt-a044B raw W&B · 4B prompted W&B
The 4B smoke passed at 137,698 tokens with 54.69 GB peak GPU allocation; its model/tokenizer, BF16 export, and all eight optimizer shards were verified. Production exposed extra gradient-buffer memory pressure, addressed by preserving DDP bucket views and using expandable allocator segments. The corrected raw attempt passed the failing step and reached step 21 before preemption. A subsequent resume shortcut exposed an eval-mode/activation-checkpointing bug; explicit training mode and a real HF-load regression test now cover it. All current bundles include both fixes.
Frequent preemptions have erased substantial unsaved work. Current bundles save after the first update of every attempt and every two minutes, and skip redundant validation at a restored production step. Run identities, complete-document batches, training objective, and token budgets are unchanged. The shorter cadence has saved the first durable checkpoints for 0.8B prompted and 2B raw. At the 21:10 UTC snapshot, the latest workers were again building/waiting after preemptions; no full-budget trial has completed. Iris retries preemptions automatically. There is no scheduled agent monitor available in this session.
Validation: 15 experiment tests, the existing 8 capacity-helper tests, implementation lint, and Pyrefly pass. The two-process numerical test checks multiple sharded-AdamW updates against a single global batch. The 2B generation canary completed on eight held-out proteins per format, but all 16 outputs hit the 256-token cap with zero valid contact pairs. Those tiny smoke checkpoints do not establish contact accuracy. Per-input timings and the canary metrics are committed.
Implementation and the current experiment record are in draft PR #348. Intermediate likelihood improvements are not a final contact-accuracy result, and the greedy AFDB diagnostic is not the FoldBench rollout-plus-resample benchmark.
timodonnell commentedon Oct 2, 2026
🤖 Follow-up at 21:16 UTC: all six current attempts now report fresh training progress, and all six production checkpoints were independently verified through in-cluster S3 reads: matching completion marker, model weights, tokenizer, and all eight optimizer shards.
Retained checkpoint tokens (raw / prompted): 0.8B: 48.83M / 3.25M; 2B: 0.289M / 50.02M; 4B: 0.289M / 0.041M. These are saved progress, not the larger pre-preemption W&B high-water marks. The corrected resume path is working, including both 4B arms. All six remain within the 48-H100 cap and continue toward 1B tokens each; no production run is complete yet.
timodonnell commentedon Oct 5, 2026
🤖 October 5 update: five of six production trials have completed approximately 1B tokens, with successful Iris/W&B terminal states and verified final checkpoints. The completed checkpoints include model/tokenizer files, BF16 inference exports, and all eight optimizer shards.
These are teacher-forced likelihoods on the same 64 held-out AFDB documents. Lower is better within a format; the tokenizations differ, so the columns should not be compared directly. Larger models improve this diagnostic. Full-budget contact-generation accuracy has not been evaluated yet.
The 2B prompted root a03 failed on October 3 at 00:13 UTC after a 600-second scalar ALLREDUCE timeout at about 79.61M tokens. No preceding OOM was recorded, and the underlying cause remains unresolved. I verified that the old root is terminal and submitted one controlled resume,
/timodonnell/exp347-qwen35-2b-prompted-1bt-a04, at batch priority on eight H100s. It restored step 1773 (78,142,147 retained tokens) and completed its first resumed update. Run identity, optimizer/data/RNG state, objective, batch size, and budget are unchanged. The other five trials need no further training resources.Updated learning curves, narrative, and 320 per-protein final-validation timing rows are committed in 62de433 on draft PR #348. The final contacts-v1 checkpoints are at step 6496; the completed prompted checkpoints are at step 22268, under their existing run-named S3 checkpoint roots. Evaluation and public checkpoint publication remain outstanding.
timodonnell commentedon Oct 5, 2026
🤖 Full-corpus preparation and periodic eval-val scoring are implemented in bcc2f0c (PR #348).
All 2,067 source shards are prepared: 3,717,047 training / 37,256 validation proteins, with 18.165B contacts-v1 / 5.214B prompted native tokens per pass. Both formats retain the same complete sequences and contact lists.
The new evaluator passed the canonical E8 legacy554 gate: R-precision 0.424377 all / 0.365988 long, within 0.0004 of reference, with all 55,400 rollouts complete and zero truncations. Metrics and measured per-protein timings are committed. This is a reference validation, not a Qwen eval-val result.
Both 4B pilot checkpoints are now being scored on the frozen 97-protein eval-val set, using 100 rollouts and unchanged exp89 resolved-pair metrics. Their persistent evaluation drivers will also process subsequent checkpoints. Contacts-v1 uses fresh offsets/order; prompted uses independent fixed-prefix samples. Native Qwen token capacity is adjusted explicitly; the longest raw target requires 34,677 context tokens, so that driver now uses 64K context and resumes completed proteins.
Working full-scale default, announced in chat after clarification remained unanswered: both 4B formats, 100B additional tokens each, evaluation every 1B, retaining the 48-H100 cap. Each phase initializes from its respective 1B pilot weights with a fresh optimizer and full-corpus cursor. This exact token budget is an operator assumption, not an additional explicit user selection.
Full-scale training is not launched yet: the two-node validation reached model staging but stalled at its first NCCL collective. Static rendezvous fixes the pods' localhost hostname resolution; a fresh test now uses explicit socket transport. Checkpoint/export/resume validation must pass before production. Twenty-one local tests pass. The existing 2B prompted pilot remains healthy past its previous failure point (120M tokens at the latest check).
Evaluation runs: contacts-v1, prompted.
timodonnell commentedon Oct 5, 2026
🤖 Both 4B 1B-token pilots now have complete eval-val baselines (97 proteins × 100 terminated rollouts each):
The differences are below the ~0.005 evaluation resolution. These are below the existing exp232 m2-p06 eval-val result (0.51980 / 0.50173); no pretrained-transfer benefit is established. The full decontaminated AFDB sequence-KNN context is 0.40715 / 0.39211, but it is not an exact null for the pilots' 512-shard subset. One prompted sample required prefix-preserving continuation; all final rollouts terminated. Eval-test was not read.
Small metric/timing tables are public in the HF bucket under
data/exp347-qwen-base-contacts/data/eval_val_pilot/, with anonymous reads verified. Persistent four-worker evaluation drivers are waiting for full-phase checkpoints under the existing raw and prompted W&B identities.The 16-GPU training/export test succeeded at 1.81M tokens, and early inference exports survived rolling checkpoint pruning. Socket transport was slower than the prior eight-GPU setup. A final native-InfiniBand resume test is underway with the corrected routable IPv4 interface; full-scale training is not yet launched. The working plan remains both formats, 100B additional tokens per arm, evaluation every 1B, within 48 H100s.
timodonnell commentedon Oct 5, 2026
🤖 Both full-corpus 4B phases are training and have verified resumable checkpoints.
Each phase targets 100B additional native tokens on the full 3,717,047-protein training corpus, starting from its corresponding 1B pilot with a fresh optimizer. This remains the announced operator default for “full scale.” Each uses 8 H100s at batch priority; 100-rollout R-precision runs on all 97 eval-val proteins every 1B additional tokens and at completion, with four evaluation workers per arm. Existing 2B pilot plus both training/evaluation arms use at most 32 H100s, below the 48-H100 limit. No eval-test reads.
At 15:25 UTC, raw had advanced to 5.05M tokens / step 34 and prompted to 123,602 tokens / step 3. Verified checkpoints were raw step 24 / 3,561,676 tokens and prompted step 1 / 48,698 tokens; each has model/tokenizer and all eight optimizer/RNG/cursor shards. The two-node engineering test restored its validation NLL exactly and saved a new checkpoint, but native InfiniBand was unavailable because the image lacks libibverbs. Socket transport underperformed the existing eight-GPU setup, so production uses one node per arm; the engineering jobs are terminal.
Baseline eval-val all/long R remains 0.14010/0.10083 (raw) and 0.14350/0.09894 (prompted), effectively tied. Step 0 on the new accuracy curves denotes those 1B-token pilot weights. Periodic BF16 exports survive recovery-checkpoint pruning, and evaluation drivers retain results after training exits. Jobs have a 90-day window with resumable state; 100B completion within that window is not guaranteed. There is no scheduled agent recovery monitor; the persistent Iris jobs execute the fixed evaluation schedule.
All 23 local tests and Ruff passed. Per-input timing artifacts and result tables are committed and published in the public HF bucket. PR #348 and the experiment README have been updated; training remains in flight.
timodonnell commentedon Oct 5, 2026
🤖 Corpus correction: the previous “full corpus” claim was wrong. That scale-up used only the AFDB component (3.717M eligible documents). Both continuations and their waiting eval drivers have been stopped and marked superseded.
I checked recent experiments #277 and #343 and audited all 9,112 source parquet footers in the existing CoreWeave east bucket. The latest ready training pool contains 235,490,905 documents: 3,963,003 native AFDB; 65,553,178 native ESM; 31,702,680 MPNN AFDB; 130,872,044 MPNN ESM; and 3,400,000 complex training documents. This is before our paired 16K/empty-contact and lineage-holdout filters. Exp292's unpublished supplement is not included.
The replacement streams and tokenizes all five sources online, preserves native/redesign cluster lineage, and supports multi-chain documents in both renderings. Source-specific admission counts are logged. The CPU path passed real-data/replay checks on all eight ranks; a 4B GPU checkpoint/recovery smoke is running before relaunch. The corrected arms start again from their original 1B pilot weights, with 100B additional tokens per format and eval-val R-precision every 1B. No cross-region corpus copy is needed.
timodonnell commentedon Oct 5, 2026
🤖 Corrected full-corpus 4B runs are now training. Both have completed optimizer updates, admitted examples from all five sources, and saved verified model/tokenizer/eight-rank recovery checkpoints.
/timodonnell/exp347-qwen35-4b-contacts_v1-corpus235m-100bt-a01/timodonnell/exp347-qwen35-4b-prompted-corpus235m-100bt-a01The source pool is 235,490,905 documents / 9,112 shards from #277 plus #343's complex training corpus, before the shared 16K, empty-contact and lineage-holdout filters. Both continuations restart from their original 1B pilot weights, use 100B additional native tokens and eight H100s each, and evaluate all 97 eval-val proteins every 1B additional tokens with 100 completed rollouts per protein.
Both persistent evaluation drivers are running: contacts-v1 R-precision, prompted R-precision. They reused the verified pilot baseline at checkpoint zero; the first new measurement is due at 1B additional tokens.
Validation: every source footer/ETag audited; 2,048 real documents admitted across an eight-rank CPU sample with exact replay; the 4B GPU recovery smoke restored validation loss exactly and committed a new eight-rank checkpoint. Local tests, changed-file Ruff and Pyrefly passed. Audit manifests and 160 per-input timing rows are committed and public in the HF bucket. The old AFDB-only scale-up is stopped and explicitly marked superseded. Updated implementation and records are in PR #348.
timodonnell commentedon Oct 19, 2026
🤖 October 19 status check: both corrected 4B full-corpus runs are stopped, with no training progress since October 5. The input correction is in place: 235,490,905 source documents across native AFDB/ESM, MPNN AFDB/ESM, and complex training data, before experiment filters.
Both checkpoint listings contain model weights, tokenizer, complete marker, and all eight optimizer shards. Neither continuation reached the first 1B-token evaluation. The 97-protein eval-val results remain the original 1B-pilot baselines: raw all/long R-precision 0.14010/0.10083, prompted 0.14350/0.09894. These are not measurements of the corrected full-corpus checkpoints.
Retained Iris logs show repeated 600-second NCCL scalar ALLREDUCE timeouts, with rank 0 one collective behind, in both 4B arms and the remaining 2B prompted pilot. The underlying cause remains unconfirmed. The training job records are absent from Iris; no replacement training jobs were found. Both evaluation CPU drivers are still running after one preemption each and waiting for new requests, despite stale W&B crashed labels.
The 2B prompted pilot also stopped at 332.0M/1B tokens; its checkpoint at 330.25M tokens is present. There was no scheduled agent recovery monitor, and these failures remained unrecovered. No restart was made during this status check: the recurring synchronization failure needs diagnosis before production recovery.
timodonnell commentedon Oct 19, 2026
🤖 Checkpoint recovery fix implemented and both 4B production resumes submitted.
The failed raw step 2077, prompted step 1638, and 2B pilot step 7486 prefixes contain all optimizer shards but incomplete rank-zero model/tokenizer uploads. The timed-out scalar collective was the checkpoint barrier. The injected storage configuration nests five s3fs attempts around botocore retries with 120-second read timeouts, permitting storage waits longer than the 600-second NCCL watchdog. The original logs do not capture which individual request first stalled.
The fix bounds each checkpoint storage operation to 90 seconds per attempt, retries timeouts at most three times, and shares a 480-second budget across checkpoint storage operations. Non-timeout errors propagate; the previous recovery pointer remains until uploads complete. Added pre-watchdog Python stack dumps, per-object storage logs, and an isolated bounded recovery-replay option. Observer state now distinguishes queued dispatches from running training. Code: 923b24d and 552492a, in #348.
Validation: 36 tests passed, including real fsspec coroutine cancellation, transient and permanent stalls, cumulative budget exhaustion, and permission-error propagation. Changed-file Ruff and Pyrefly pass. The bounded put/read/marker/delete path passed against actual CoreWeave S3; a separate 120-upload probe also completed. GPU validation remains pending allocation.
Production roots, eight H100s each at batch priority:
/timodonnell/exp347-qwen35-4b-contacts_v1-corpus235m-100bt-a02/timodonnell/exp347-qwen35-4b-prompted-corpus235m-100bt-a02They retain the original run/checkpoint identities, saved optimizer/RNG/data cursors, 235.49M-source corpus, 100B additional-token budgets, and every-1B-token eval-val schedule. Both eval drivers remain active. The diagnostic roots were canceled before allocation and used no GPUs.
Current blocker: all 256 east H100s are allocated. Both production resumes are queued, with no new training progress yet. RNO has available capacity, but moving this experiment's storage traffic across regions requires the repository's explicit >10GB approval; that question is pending with the user. No cross-region copy was performed.
timodonnell commentedon Oct 21, 2026
🤖 Live status checked October 21, 2026, around 13:51 UTC.
Both 4B continuations resumed on the corrected 235,490,905-source-document corpus and made progress after the October 19 checkpoint-I/O fix.
The new checkpoint path has successfully saved complete checkpoints. I independently verified raw step 4620 and prompted step 11981: matching resume/complete markers, model and tokenizer, all eight optimizer shards, 20 objects totaling about 50.49 GB each.
The raw failure is a different issue: rank 0 reports an NCCL CUDA out-of-memory error in
scaled_loss.backward()during its first update after restoring step 4620. It had experienced 30 preemptions. The prompted job has experienced 45 preemptions and is currently running on eight H100s; its latest reported peak PyTorch allocation is 71.10 GB. The precise source of the raw restart memory peak remains unconfirmed, so I did not blindly retry the same failing first update.Both periodic eval-val CPU drivers are live. Neither continuation has reached its first 1B additional-token evaluation. The only completed 97-protein R-precision measurements remain the original 1B-pilot baselines: raw 0.1401 overall / 0.1008 long-range; prompted 0.1435 / 0.0989. These are not scores for the new continuation checkpoints.
No cross-region transfer was performed. The raw failure disposition and current observations are persisted in the experiment state. There is still no scheduled agent recovery monitor; Iris handles preemptions but the terminal raw failure required this status check to surface.
Raw W&B · Prompted W&B
timodonnell commentedon Oct 21, 2026
🤖 The October 21 restart-memory repair has passed real eight-H100 training and checkpoint validation for both 4B formats.
The raw run previously failed on its first backward pass after restoring step 4620. The corrected run completed that exact next update (step 4621), saved model/tokenizer plus all eight optimizer shards, and continued through subsequent updates. Prompted likewise resumed step 12156, committed step 12157, and continued. The replacement Iris roots are
/timodonnell/exp347-qwen35-4b-contacts_v1-corpus235m-100bt-a03and/timodonnell/exp347-qwen35-4b-prompted-corpus235m-100bt-a03.Commit
d2836a26synchronizes the first microbatch of the first update after DDP construction, establishing shared gradient buffers before accumulating the remaining microbatches. It preserves the global token-mean gradient, optimizer state, RNG/data cursors, learning-rate schedule, and token budgets. A distributed regression reconstructs DDP with restored Adam state and verifies global gradients, updates, and buffer sharing. All 37 tests, Ruff, and Pyrefly pass. First-update PyTorch peak allocations were 76.55 GB raw and 55.62 GB prompted; these are different batches, not a controlled format comparison.Interim eval-val scoring is underway for preserved raw step 4620 / 498,970,942 additional tokens and prompted step 12051 / 450,586,475 additional tokens, with twelve single-H100 shards per checkpoint. Both checkpoints passed the one-protein, 100-rollout gate with no unfinished samples. The canonical 97-protein evaluation, exp89 metrics, and sampling settings remain unchanged; eval-test is untouched. Results will be reported after completeness validation. All storage and compute remain in CoreWeave east, within the existing 48-H100 ceiling.
timodonnell commentedon Oct 21, 2026
🤖 Fixed the restart memory failure and completed both interim eval-val measurements.
The first microbatch after DDP/optimizer restoration now establishes shared gradient bucket storage before further accumulation. Both 8-H100 runs passed the previously problematic first update and saved complete model/tokenizer plus eight optimizer shards. Raw continues as a03; prompted a04 resumed after a short pause for evaluation capacity and committed a fresh step-12219 checkpoint, independently verified at 14:47 UTC. Both retain the 235.49M-source-document pool and 100B-additional-token targets.
Each measurement covers all 97 eval-val proteins × 100 completed rollouts, with zero unfinished samples. One prompted sample required prefix-preserving continuation. Relative to the 1B-token pilots, raw improved +0.00471 overall / +0.00704 long-range; prompted improved +0.00889 / +0.00844. Accuracy remains low, and long-range scores are tied within the approximately 0.005 evaluation resolution. The checkpoints have unequal additional token/protein exposure. Eval-test remains untouched.
Results are logged in raw W&B and prompted W&B. The regular schedule remains every additional 1B tokens.
PR #348 contains the fix, 37 passing tests, complete metrics/timings, dispatch/provenance records, viral strata, and updated analysis. The small result artifacts are also anonymously readable under
data/exp347-qwen-base-contacts/data/eval_val_interim_20261021/in the public HF bucket; all 33 files were checksum-verified after publication.timodonnell commentedon Oct 22, 2026
🤖 Status checked October 22, 10:43 a.m. EDT: both 4B training jobs are currently stopped.
Raw a03 failed October 21 at 1:17 p.m. EDT after eight preemptions; prompted a04 failed at 1:30 p.m. after nine. Both final attempts hit a 600-second NCCL synchronization timeout on the same three-element collective after restart. The captured failure logs contain no out-of-memory error. The shared cause remains unresolved, so these correlated failures have not been blindly relaunched.
Both saved checkpoints were independently rechecked: completion marker and recovery pointer match; all eight optimizer shards, indexed model weights, and tokenizer are present. Both periodic evaluation CPU drivers remain live. Neither training arm reached the next 1B-token milestone; no newer evaluation request exists.
Latest completed eval-val results remain 0.14481 overall / 0.10787 long-range for contacts-v1 at +498.97M tokens, and 0.15239 / 0.10738 for prompted at +450.59M. Each covers all 97 proteins × 100 completed rollouts. Yesterday's successful recovery/update/checkpoint verification was real, but did not establish reliability across subsequent preemptions.
timodonnell commentedon Oct 22, 2026
🤖 Applied the startup repair and submitted both production recoveries.
The two failed attempts printed
WANDB_RUNbut never updated the run-registration object or logged their restored step. The registration objects' last writes remained at 16:47:37 UTC (raw) and 16:45:52 UTC (prompted), before the failed 17:06/17:19 restarts. Seven ranks had reached the first three-element token-count collective while rank zero had not entered the loop diagnostics. This places the stall in the unbounded buffered S3 metadata write after W&B initialization. All 16 optimizer shard metadata records agree with their checkpoint counters and rank assignments.Commit 7d9849d routes startup metadata and evaluation-request writes through cancellable
pipe_file, with a 60-second total budget and 20-second operation deadlines, and starts stack diagnostics before startup bookkeeping. The launcher now supplies the explicitly required CoreWeave credentials with redacted records. The valid existingcwcredential profile passed real checkpoint reads and metadata write/read checks. 37 tests, Ruff, and Pyrefly pass.Accepted recovery roots:
/timodonnell/exp347-qwen35-4b-contacts_v1-corpus235m-100bt-a04, from step 5229 / 574,648,070 additional tokens./timodonnell/exp347-qwen35-4b-prompted-corpus235m-100bt-a05, from step 12538 / 476,908,170 additional tokens.Both remain capacity-pending in east as of 19:54 UTC (3:54 p.m. EDT); renewed GPU progress has not yet been verified. All 256 east H100s were occupied. The two pending jobs retain their optimizer/data/RNG state, run identities, 100B-token targets, and 1B-token evaluation cadence. Diagnostic-only submissions were retired before GPU admission.
RNO2A showed 49 free H100s, but the existing source corpus and two recovery checkpoints total 579.83 GB. Cross-region use, including subsequent checkpoint traffic, requires explicit approval under AGENTS.md; that question is pending. No cross-region bulk transfer or shared-cluster modification was performed. PR #348 contains the fix and current execution record.
timodonnell commentedon Oct 22, 2026
🤖 Both full-corpus 4B runs are training again on RNO2A, with all new checkpoints written in RNO2A.
/timodonnell/exp347-qwen35-4b-contacts_v1-corpus235m-100bt-a05, accepted 20:35:01 UTC. First verified local checkpoint: step 5230 / 574,764,731 additional tokens./timodonnell/exp347-qwen35-4b-prompted-corpus235m-100bt-a06, accepted 20:36:50 UTC. First verified local checkpoint: step 12539 / 476,967,387 additional tokens.Both checkpoints were independently checked against their completion/resume markers, all eight optimizer shards, five model shards, and tokenizer: 20 objects / 50.49 GB per run. Their new root is
s3://marin-rno2a/MarinFold/exp347_qwen_base_contacts/checkpoints/<run>/step-<N>/. CoreWeave reportsBucketRegion=RNO2Afor this bucket. Neither original east checkpoint root has received new objects since migration.Commit
990e3641routes recovery checkpoints, BF16 evaluation exports, requests/results, and timing outputs by compute region.--relocate-fromrestores the existing full model/optimizer/RNG/data state once; subsequent restarts prefer the new local checkpoint. The immutable source-corpus URIs remain unchanged, using the approved cross-region reads and LOTA cache, so the catalog hash and URI-seeded data order are preserved. The 235.49M-source-document pool, 100B additional-token targets, eight ranks per run, and W&B identities remain unchanged.The prior east training and evaluation roots were confirmed stopped before replacement. Both periodic evaluation drivers are live as
a02in RNO2A and place their GPU children there. Eval-val remains every 1B additional tokens. The October 21 all/long R-precision values remain the latest completed measurements: contacts-v1 0.14481 / 0.10787, prompted 0.15239 / 0.10738. No new R-precision point is claimed for this recovery.Validation: 39 tests, Ruff, and Pyrefly pass. A separate eight-H100 recovery replay completed two updates and two full local saves, then exited successfully at step 5231 with zero failures/preemptions. Both production runs subsequently completed their own restored update and local checkpoint. Total ongoing training allocation is 16 H100s, at batch priority.
Contacts-v1 W&B · Prompted W&B · Implementation PR #348
timodonnell commentedon Oct 23, 2026
🤖 Both 4B training arms are running again on RNO2A after the storage credential refresh. All new checkpoint and evaluation writes remain in
marin-rno2a.The previous attempts stopped with
InvalidAccessKeyIdafter approximately five hours. The refreshed credentials passed direct read/write checks, and the prior local model/tokenizer/optimizer checkpoints were independently verified before launch. No training-code change was needed./timodonnell/exp347-qwen35-4b-contacts_v1-corpus235m-100bt-a06/timodonnell/exp347-qwen35-4b-prompted-corpus235m-100bt-a07Each new checkpoint has matching completion/resume markers and all 20 objects: model shards, tokenizer, and eight optimizer shards, approximately 50.49 GB. Both jobs use eight H100s at batch priority. Three scheduler preemptions delayed startup; Iris retried the same roots automatically. At 14:53 UTC, both were advancing beyond those saves: raw step 7221 / 835,605,424 tokens and prompted step 15658 / 680,284,817 tokens, with no application failures.
Both persistent eval-val drivers are running as
a03on RNO2A, with their GPU children also pinned there. The 100B additional-token targets, source corpus, data order, run identities, and evaluation cadence remain unchanged. Next periodic evaluation: 1B additional tokens per arm. The October 21 R-precision values remain the latest completed measurements; no newer scores are claimed.contacts-v1 W&B · prompted W&B
timodonnell commentedon Oct 26, 2026
🤖 October 26 update: both 4B training runs are progressing on RNO2A.
The training roots remain raw a06 and prompted a07, each on eight H100s. Both full checkpoints have matching completion/resume markers, all model/tokenizer files, and all eight optimizer shards in
marin-rno2a.Periodic evaluations had failed on the first 1B request after relocation. I fixed two causes: the RNO driver requested an invalid federation hop back to its own controller, and the worker still enforced the old east-only checkpoint bucket. Regional validation is now shared by both launch paths and the worker. All 45 tests pass, including six locality regression cases; Ruff and helper/launcher/driver Pyrefly checks pass.
Replacement evaluation drivers raw a06 and prompted a05 are running. All eight GPU workers loaded local 1B-token exports and emitted per-protein scores by 16:07 UTC. They will process the saved raw 1B–5B and prompted 1B–2B checkpoints in order, then resume the 1B-token cadence. Training was not restarted. All new checkpoints and evaluation outputs stay in RNO2A.
No new full 97-protein aggregate has completed yet. Latest completed all/long R-precision remains the October 21 points: contacts-v1 0.14481 / 0.10787 at +0.499B tokens; prompted 0.15239 / 0.10738 at +0.451B. The evaluation contract remains 97 eval-val proteins × 100 terminated rollouts with saved per-protein timings; eval-test remains untouched.
timodonnell commentedon Oct 30, 2026
🤖 October 30: nine complete periodic eval-val results are now available.
Every point covers all 97 eval-val proteins × 100 terminated rollouts, with no unfinished samples. The downloaded 873 timing rows and 17,460 metric rows reproduce the aggregates. Complete checkpoint/worker provenance and viral splits are saved in
data/eval_val_periodic; the learning curve is updated. Eval-test was not read. Prompted leads at 3B native tokens, but different protein exposure prevents isolating a causal format effect. Raw 5B and 6B are tied within ~0.005. Both remain below exp232 m2-p06 (0.51980 / 0.50173); an exact sequence-KNN null for the expanded corpus is not yet available.Training recovery: both roots failed October 27 at 12:35–12:37 UTC on
Account is write suspended for this availability zoneduring RNO checkpoint uploads. Final observed progress was 6.709B raw / 3.155B prompted. Complete recoverable checkpoints at steps 44,293 / 67,797 retain 6.705B / 3.153B tokens; both 20-object checkpoints were independently verified today.RNO PUT/GET and multipart UploadPart now pass. Replacement raw a07 and prompted a08 were accepted at 20:16 / 20:18 UTC and are currently Kueue-gated awaiting GPUs. No resumed optimizer updates are claimed yet. They preserve the original optimizer/data/RNG state, 100B targets, local checkpoint writes, and 1B evaluation cadence. Both periodic evaluation drivers remain live and have completed all existing requests. No model-training code changed for this recovery.
timodonnell commentedon Oct 30, 2026
🤖 TPU fork preparation is committed in 4ad6365 (PR #348). The GPU parents remain running: raw a07 and prompted a08 have resumed and produced complete new RNO checkpoints, observed at 6.712B and 3.163B additional tokens. Their existing evaluation drivers continue independently.
The proposed TPU forks retain both 4B formats under new run identities, preserving FP32 weights, Adam moments/step, eight logical data cursors, absolute token counts, and the remaining 100B-token schedules. The regional mirrored reader is implemented and tested: it reproduces rows and cursor state across an epoch boundary after the original test files are deleted. All 54 tests, Ruff and Pyrefly pass.
The current CUDA trainer cannot run unchanged on TPUs. MaxText has the Qwen3.5 hybrid architecture, but the 4B adapter, full-state converter, forward/gradient/update parity, TPU save/restore checks, and regional evaluation integration still need implementation and validation. No TPU training has launched. Candidate east5 placement starts with one validation slice before admitting at most two eight-unit production slices.
The committed transfer inventory totals 579.87 GB: the complete 235.49M-document raw corpus plus two full training checkpoints and small validation/catalog artifacts. A proposed 590 GB ceiling covers one copy into us-east5. The existing GCS exp232 cache uses the wrong tokenizer. All ongoing TPU corpus reads, checkpoints and evaluation outputs would remain under
gs://marin-us-east5/protein-structure/MarinFold/exp347_qwen_base_contacts/. The observed checkpoint paths are rolling and are not frozen parents; exact parent snapshots must be selected at fork time.No bulk copy has started. Explicit approval for this new CoreWeave-to-GCP transfer is pending under the root AGENTS.md >10GB rule. The earlier RNO transfer approval did not specify this route. No new R-precision point is claimed by this preparation.