[timodonnell]

Fine-tune pretrained Qwen3.5 base models (0.8B, 2B, 4B) to predict protein contacts, comparing native contacts-v1 documents with natural-language instructions, ordinary amino-acid sequences, and one-based contact pairs. Retain the native tokenizer and update all text-model weights. Tracks #347. Five of six 1B-token pilots completed; the partial 2B prompted pilot remains inactive.

Both 4B arms continue on the audited 235,490,905-source-document pool toward 100B additional tokens each, with eval-val scoring every 1B tokens. The source count precedes paired length/empty/lineage filtering. Each arm uses eight H100s; periodic evaluation uses up to four more per arm. All new checkpoints, BF16 exports, evaluation results, and timings stay in RNO2A (marin-rno2a). Approved corpus reads preserve immutable east URIs and LOTA caching, maintaining catalog hashes and seeded data order.

All thirteen periodic eval-val milestones are complete (raw 1B–9B, prompted 1B–4B). Each requires 97 proteins × 100 terminated rollouts, unchanged exp89 resolved-pair metrics, saved per-protein timings, and checkpoint/code provenance. Latest results:

Format	Additional tokens	Step	All-range R	Long-range R
contacts-v1	9,000,147,334	58,458	0.27164	0.23281
prompted	4,000,013,818	86,491	0.24773	0.21288

Raw improves by 0.03099 / 0.03481 from 6B to 9B. At matched 4B native tokens, contacts-v1 scores 0.21607 / 0.17332, so prompted leads by 0.03166 / 0.03956. Different protein exposure prevents a causal format claim. Both remain below exp232 m2-p06 (0.51980 / 0.50173); an exact sequence-KNN null over the expanded corpus has not been computed. Eval-test remains untouched. Complete periodic artifacts contain 1,261 timing rows, 25,220 metric rows, 52 worker provenance files, and viral splits.

Recovery preserves optimizer/data/RNG state and run identities, initializes DDP gradient bucket views on the first microbatch, and bounds checkpoint/startup I/O. Regional evaluation children inherit the driver's controller rather than requesting a federation hop to themselves. Both launch paths and the worker enforce compute-local checkpoint and output paths. The corrected evaluation drivers completed their backlogs and remain live.

Both training jobs stopped again November 1 at 11:20–11:21 UTC because CoreWeave suspended account writes in RNO2A. Last logged progress was 9.111B raw / 4.023B prompted; full local checkpoints retain 9.108B / 4.022B tokens. On November 2, all 20 objects in each checkpoint and matching completion/resume markers were independently verified. Regional PUT/GET and a 5 MiB multipart UploadPart probe pass. Replacement raw a08 and prompted a09 were accepted at 14:30:26 / 14:32:33 UTC, preserving full training state and all scientific settings. Both replacements resumed optimizer updates and committed new full local checkpoints, independently verified at raw step 59,223 / 9,107,834,590 tokens and prompted step 87,226 / 4,021,881,608 tokens. Both Iris and W&B report running. Both evaluation drivers remain live with no backlog.

The requested TPU forks are in preparation; no TPU job or bulk transfer has launched. A new mirrored corpus reader preserves original URI-based shuffle identities and eight logical data cursors while resolving a verified regional copy. It rejects incomplete maps and changed source/destination identities. The candidate backend is MaxText's Qwen3.5 implementation; the 4B adapter, full Adam-state conversion, numerical parity checks, and TPU save/restore validation remain to implement. GPU parents continue independently. TPU forks will have distinct W&B/checkpoint identities and retain the remaining token schedule and 1B-token eval-val cadence.

The committed transfer inventory estimates 579.87 GB for the complete raw corpus, two FP32 model/Adam checkpoints and validation artifacts, with a proposed 590 GB approval ceiling. Target storage is gs://marin-us-east5/protein-structure/MarinFold/exp347_qwen_base_contacts/; ongoing training and evaluation I/O must remain in that region. Fresh approval is pending under the repository's >10GB cross-region rule. Observed rolling checkpoints are size/configuration evidence; durable parent snapshots must be selected at fork time. Proposed admission is one validation slice, then at most two eight-unit production slices after all gates pass. The 4B TPU backend has not been validated.

Validation: the implementation previously passed 54 tests, Ruff and Pyrefly, including regional replay across an epoch boundary after removing original source files, missing/duplicate/mismatched mirror rejection, restored-Adam gradient/update equivalence, BF16 exports, resume markers, and regional-path regressions. This update adds result artifacts, documentation, and run history without changing training code. All four new aggregates recompute from the full metric tables, and 35 new or updated public artifacts pass anonymous SHA256 verification. The updated result tables, learning curve, 18-page summary, and prior transfer inventory are published to the public HF bucket. These checks validate the existing GPU implementation and mirrored reader; they do not establish TPU correctness. The existing E8 evaluator gate reproduced all/long R=0.42438/0.36599 on 554 proteins.