New record submissions for review (#1583, #1584, #1585)

[codemath3000]

@valerio-oai @0hq @cocohearts @openai/parameter-golf-team Hi! I've submitted three related record PRs applying systems-level performance optimizations (fused Muon kernel, batched EMA, loader prealloc) to different base stacks, each improving val_bpb over the respective baseline:

• Record: SP8192 + Systems Optimization — val_bpb 1.0801 (3-seed mean) #1583 — on PR Record: SP8192 + 3-Layer Recurrence + Parallel Residuals + QK-Gain 5.25 + Legal TTT — val_bpb 1.0810 (3-seed mean) #1493 (current SOTA), val_bpb 1.0801 (from 1.0810)
• Record: Improved Parallel Residuals + Systems Optimization — val_bpb 1.0752 (3-seed mean) #1584 — on PR Record: ImprovedParallelResiduals, 1.0758 BPB / 2.7789 nats, -0.0020 BPB / -0.0052 nats vs PR #1523 #1529, val_bpb 1.0752 (from 1.0758)
• Record: Casefold Tokenizer + Parallel Residuals + Systems Optimization — val_bpb 1.0639 (3-seed mean) #1585 — on PR Record: Custom Casefold Tokenizer — 1.0668 BPB #1578, val_bpb 1.0639 (from 1.0668)

They're submitted against multiple bases so a ready-to-merge option exists regardless of how the pending PRs are resolved.

This is my first submission to the challenge, so I apologize in advance if I've gotten anything wrong — I've done my best to follow the submission guidelines. Happy to address any issues. Thank you for taking the time to review these, I know you're busy!

[msisovic]

Hey @codemath3000, submissions need to be at least 0.005 nats better than their respective baseline. Keep that in mind before spending compute on capturing multiple runs.

[codemath3000]

Thank you for bringing this up, @msisovic! Per the official rules, "For submissions that improve speed through systems optimization without changing the ML, this requirement [0.005 nats] is waived." This submission changes only systems-level code (kernel fusion, batched ops, memory preallocation) without altering model architecture, optimizer logic, loss function, or any hyperparameter, so it fits under this criterion, waiving the 0.005 nats threshold.

Additionally, #1585 does meet the 0.005 nat threshold relative to its associated submission regardless.

Furthermore, #1584 meets the 0.005 nat threshold relative to the current leaderboard SOTA, which is how virtually all other submissions measure nats. From what I can tell at least, even submissions that are derived from/based on another submission typically measure nats relative to the current leaderboard SOTA at the time of posting, not relative to the parent submission that it is based off of. For example, #1477 (an official, merged, approved PR) is based on #1413 and met the 0.005 nats threshold relative to the leaderboard SOTA at the time, meaning it got merged, even though it almost certainly did not meet the threshold relative to #1413, which had not been merged at the time #1477 was submitted.

[msisovic]

Ah, I wasn't aware of the exception for systems optimizations, cool!