MTP (Multi-Token Prediction) speculative decode for Qwythos / Qwen3.5 NextN GGUF

[ai-hpc]

Summary

Track implementation and optimization of Multi-Token Prediction (MTP / NextN) speculative decoding in sparkinfer for Qwythos-9B MTP GGUFs. Reference architecture: llama.cpp draft-mtp (PR #22673).

Current status (RTX 5090, Qwythos-9B-Claude-Mythos-5-1M-MTP-Q4_K_M.gguf)

Correctness

Check	Result
SPEC_AGREE (MTP generate vs AR generate)	100% — greedy output identical
DRAFT_TOP1 (MTP draft vs main verify)	~58%
Generate acceptance (real prompt)	~68% (54/80 extra drafts)
Trunk vs llama.cpp	MTP head loads; trunk path unchanged

Decode throughput (ctx=0)

Engine	Mode	tok/s
sparkinfer	AR (qwen3_gguf_bench)	~303
sparkinfer	MTP (SPARKINFER_BENCH_MTP=1, draft_max=3)	~176 (49% acceptance on synthetic)
llama.cpp e8f19cc	AR (llama-bench tg64)	225
llama.cpp	MTP (llama-cli --spec-type draft-mtp --spec-draft-n-max 3)	262 (~1.22× vs AR on same prompt)

MTP bench is slower than AR on synthetic tokens because draft+verify overhead exceeds gains at ~49% acceptance. Real prompts show higher acceptance (~68%) and llama.cpp MTP shows ~22% speedup on a text prompt.

What is implemented (WIP branch)

• Load MTP NextN head (blk.32) from GGUF; trunk = block_count - nextn_predict_layers
• qwen35_mtp.cpp: eh_proj → gated attn + FFN → shared_head_norm → lm_head
• Speculative loop in Qwen35Model::generate() with GDN snapshot + KV truncate on reject
• qwen3_gguf_mtp_check: SPEC_AGREE + DRAFT_TOP1 + KL(main‖mtp)
• bench/scripts/mtp_accuracy.sh wrapper
• Env: SPARKINFER_MTP, SPARKINFER_MTP_DRAFT_MAX, SPARKINFER_MTP_FAST, SPARKINFER_MTP_ADAPTIVE, SPARKINFER_BENCH_MTP

Optimization plan (priority)

P0 — Throughput (target: MTP ≥ AR on ctx=0)

1. Batched verify — two-phase decode like llama.cpp (verify N drafts in one graph pass)
2. CUDA graph for MTP draft head — capture 1–3 draft steps; disable graph only on verify rollback
3. Reduce per-draft sync — async argmax chain where safe

P1 — Acceptance (target: ≥70% steady-state @ draft=3)

4. Trunk hidden capture — confirm pre-final-norm h+routed matches llama.cpp hnorm input
5. Chained hidden — ✅ draft 2+ uses mtp_chain_h output (done)
6. FastMTP tuning — default full vocab; optional SPARKINFER_MTP_FAST=32768 for speed (hurts acceptance on Qwythos 248k vocab)

P2 — Production

7. Server: auto-enable MTP when NextN GGUF loaded + SPARKINFER_MTP=1
8. Bench parity — SPEED-Bench style eval vs llama-server MTP metrics
9. int8 KV for MTP head — tune GQA-4 hd256 flash-decode splits

P3 — Not in scope yet

• Native function calling (<tool_call> parsing)
• Batched prefill integration with MTP (PR perf(qwen35): route Qwythos generate/TTFT through batched prefill #490)

Acceptance criteria

• SPEC_AGREE == 1.0 on eval prompt (held-out seed)
• MTP decode tok/s ≥ AR tok/s @ ctx=0 on RTX 5090 (same GGUF)
• Draft acceptance ≥ 60% on eval corpus with draft_max=3
• accuracy.sh trunk top1/KL vs llama.cpp unchanged with MTP GGUF loaded

References

• llama.cpp MTP: --spec-type draft-mtp --spec-draft-n-max 3
• Qwythos MTP GGUF: empero-ai/Qwythos-9B-Claude-Mythos-5-1M-MTP-Q4_K_M.gguf
• sparkinfer dashboard Qwen3.5 baseline: ~220 tok/s llama.cpp vs frontier sparkinfer AR

[ai-hpc]

P0 (batched verify) landed on the PR branch — MTP 319.9 tok/s vs AR 304.7 tok/s @ ctx=0 on RTX 5090 (1.05×), SPEC_AGREE = 1.0000 against true AR for both the batched path and the rewritten sequential fallback. llama.cpp MTP on the same box: 262 tok/s.

Acceptance criteria status:

• SPEC_AGREE == 1.0 on eval prompt — note the original gate was vacuous (generate() overrode set_mtp_decode(false) from env); now fixed and genuinely passing
• MTP decode tok/s ≥ AR tok/s @ ctx=0 (319.9 vs 304.7)
• Draft acceptance ≥ 60% with draft_max=3 — 62.5% on the synthetic bench (warm draft-head KV helped: 55% → 62.5%); real-prompt acceptance should be higher
• trunk top1/KL vs llama.cpp unchanged — AR path untouched (bench identical at 304.7); full accuracy.sh run vs llama-server still to do

P0 items now done: batched verify ✓, CUDA graph for the whole batched pass ✓ (drafts still eager — chain-graphing is the next ~0.1ms), reduced syncs ✓ (one host sync per iteration). P1 remains: MTP-head KV fidelity + trunk-hidden capture check for acceptance; FastMTP trim for the ~0.45ms/draft LM head.