Non-record: 30 experiments across 13 architectures (MLA, Pause Tokens, Eigenweight, 9 exotic ideas)

[nnm2602]

Summary

Systematic exploration of 30 experiments across 13 distinct architectural ideas for parameter-efficient language modeling, run on 1x H100 with 10-min wallclock cap each.

Best result: MLA latent=128 achieves 1.3223 BPB (only +0.013 behind baseline) with 4x KV compression.

Key findings:

• MLA (Multi-Head Latent Attention): Compressing K,V through a shared latent bottleneck nearly matches baseline at zero step-time overhead. Validates DeepSeek V2's insight at small scale.
• Pause tokens: Inserting 4 learned dummy tokens every 64 positions beats baseline at matched steps (1.3424 vs 1.3465 at step 1200). Zero architectural change, 2,048 extra params.
• Eigenweight rank sweep: Clean Pareto curve — rank 64 (4.3x compression, 1.50 BPB) → rank 256 (1.1x, 1.36 BPB). MLP rank matters more than attention rank.
• Depth recurrence: Less is more on 1 GPU — 2-layer x2 (1.3226) beats 3-layer x2 (1.3324) and 2-layer x3 (1.3399).
• 9 exotic architectures: SIREN weight generation, neural cellular automata, Hopfield energy LMs, hypernetworks, tensor MPS, seed models, communicating agents, basis sharing, universal transformer + ACT — all interesting failures with documented lessons.

Submission track

Non-record (1x H100, not 8x). Emphasis on breadth of exploration and architectural insights rather than SOTA BPB.

Test plan

• All 16 experiment scripts parse without errors
• Results are reproducible on H100 with documented run commands
• submission.json matches reported metrics

🤖 Generated with Claude Code