What blocks --mtp together with --ssd-streaming? (data: it would matter most on disk-bound rigs)

[giannisanni]

ds4.c refuses the combination with "--ssd-streaming is not compatible with --mtp yet". I'd like to understand what the actual blocker is, because measurements on consumer hardware suggest the combination is where MTP would pay off the most.

Setup: RTX 4060 Ti 16GB, 32GB RAM, Linux/CUDA, DeepSeek V4 Flash q2 (81GB GGUF) streaming from a PCIe 3 NVMe (1.5GB/s cold reads).

Measured:

• generation: 2.2 t/s (disk-bound: ~675MB of routed experts fetched per token)
• prefill on a ~1k token prompt: 28.7 t/s, i.e. ~13x cheaper per token, because batching amortizes each expert load across the batch

MTP verification batches 2-3 tokens per forward pass, so on a disk-bound rig every accepted draft should amortize expert fetches the same way prefill does. Back of the envelope that turns 2.2 t/s into 4-6 t/s, a much bigger relative win than the same feature on machines where the model is RAM-resident.

Is the blocker that the streaming selected-expert path only handles single-token decode expert sets, or something deeper (draft/verifier KV interaction with the streaming graph)? If it's the former I'd be happy to attempt it - I have a consumer test rig (soon dual-GPU) and time to validate against the official logits fixtures.

Related: #494 (small toolkit-compat PR from the same test effort).

[giannisanni]

Follow-up: I did a deep read of both paths to answer my own question about where the incompatibility actually lives. Sharing in case it helps whoever picks this up.

The streaming expert layer is already batch-capable. cuda_stream_selected_cache_begin_compact_load and routed_moe_launch are token-count-generic: streaming prefill already drives them with the deduplicated union of a whole chunk's routing sets (ds4_gpu_stream_expert_cache_prepare_selected_batch), and the kernel checks slot_count >= n_tokens*n_expert generically. Nothing in the streaming cache machinery assumes one token.

What is single-token-bound is only the decode-side host orchestration: the selected_ids[DS4_MAX_EXPERT_USED] stack buffers in metal_graph_decode_cuda_selected_load (and the readahead/async variants), the 1-token decode graph tensors (router_selected, routed_gate/mid/down), and the n_tokens > 1 gate that makes batch selected-load prefill-only.

On the MTP side, the verifier is already prefill-shaped - metal_graph_verify_suffix_tops runs all drafts through metal_graph_encode_layer_batch in one pass. So for Metal, MTP verification + streaming looks like routing that batch through the streaming batch selected-load (union of the draft tokens' experts) rather than assuming resident weights, plus deciding what KV frontier rollback means for streaming state.

For CUDA the gap is bigger: as far as I can tell the drafter/verifier/spec-frontier orchestration is Metal-only today, so CUDA needs the MTP orchestration ported before any of the above applies.

If you'd take a PR series for the Metal-side compatibility (streaming-aware verify path) or want the CUDA MTP port attempted, I'm interested - consumer disk-bound rigs are where the accepted-draft batching pays off most (my measurements: per-token cost in batched verify-shaped passes is ~13x cheaper than single-token decode when experts stream from disk).

[giannisanni]

Update: we tested the combination empirically by bypassing the guard behind an env var (DS4_MTP_STREAMING_UNSAFE=1) on the CUDA backend.

Result: it works, and strict mode is correct.

• With DS4_MTP_STRICT=1 (exact N=2 verifier), greedy output under --mtp --ssd-streaming is token-identical to plain greedy streaming on the same prompt. No crashes, no bad reads, coherent long-form output.
• Default margin mode also runs fine but diverges textually from greedy, which matches its documented approximate fast-accept semantics - not a streaming issue as far as we can tell.
• This makes sense given the architecture: the streaming selected-load path is already token-count-generic (prefill uses it with unions), and the verify batch (2-3 tokens) passes the existing n_tokens > 1 gate into the batch selected-load.

Performance today is unfavorable on our rig, for reasons that are hardware-specific: 0.93 t/s strict-MTP vs 2.72 t/s baseline (RTX 4060 Ti 16GB, experts streaming from a bandwidth-saturated 1.5GB/s PCIe 3 NVMe). Two causes: (1) each verify cycle loads the union of the draft tokens' experts, i.e. 2-3x bytes per cycle on a drive that has no bandwidth headroom; (2) the 3.8GB MTP GGUF goes resident on a card with ~1GB VRAM to spare, adding memory pressure. Both flip on faster storage - we have a Gen5 drive (14.9GB/s) arriving next week and will post comparative numbers.

Repro: DS4_MTP_STREAMING_UNSAFE=1 DS4_MTP_STRICT=1 ./ds4 -m ds4flash-q2.gguf --cuda --ssd-streaming --mtp DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf --mtp-draft 2 --temp 0 --nothink

Open edges we know we haven't validated: long-generation stability, behavior with the VRAM streaming expert cache enabled (it auto-disabled on our 16GB card), and margin-mode acceptability. If you can share what specifically motivated the guard, we'll target validation there - happy to turn this into a PR that narrows the guard (e.g. allow with strict verify) once the edges are covered.

[giannisanni]

For anyone landing here, related work in flight that we found after posting:

• Add DSpark speculative draft runtime #480 / DSpark B2 rejection sampling + adaptive block sizing #482 (DSpark speculative runtime): complementary approach; nothing there interacts with --ssd-streaming yet as far as we can tell, while this issue is specifically about the streaming path. The two could compose.
• cuda: fall back to pinned host memory when the model arena runs out of VRAM #487 (pinned-host fallback for non-routed weights): directly relevant to the VRAM pressure we hit above - the 3.8GB MTP support model going resident is what squeezes a 16GB card. We plan to test that branch together with the streaming+MTP combination.
• CUDA: batch gate/up/down uploads for selected expert cache misses #460 and [3/N] add prefetch support for CUDA backend : running ds4 for any GPU with cache (2.75 x faster!) #402 touch the same expert upload/prefetch hot path and would interact with any perf work that comes out of this.

[giannisanni]

Full benchmark matrix for the combination (Flash q2, RTX 4060 Ti 16GB, PCIe 3 NVMe at 1.5GB/s, 100 tokens, greedy, ctx 2048):

config	generation
baseline, no MTP	2.65 t/s
strict draft-1	2.66 t/s (parity: drafting itself is free)
strict draft-2	1.25 t/s
strict draft-3	0.99 t/s
margin draft-2	crash: "MTP verifier failed" after moe arena OOM
margin draft-4	0.85 t/s (verify OOMs every cycle, recovered by fallback)

We root-caused the crash. The verifier can issue single-position ffn batch encodes, and metal_graph_cuda_stream_prefill_batch_selected_load() skips when n_tokens <= 1, so the routed MoE falls through to resident full-expert range allocation (~1.8GB/layer), which cannot fit under streaming. Debug trace that pinned it:

ds4-debug: sel-gate skip layer=2 ntok=1 slots_exp=1 model=1 bsel=1
ds4: CUDA model arena alloc failed for moe_gate (1792.00 MiB chunk): out of memory

Fix is one line (allow n_tokens == 1 through the selected load): #497. After it, margin draft-2 runs clean (1.32 t/s), margin draft-4 stops OOM-looping, strict stays token-identical to greedy, baseline unaffected.

On this drive the higher draft depths lose because each verify cycle loads the union of the drafted tokens' experts on a bandwidth-saturated disk, while strict draft-1 running at exact parity shows the speculative machinery itself adds no overhead. We expect the depth-2/3 numbers to flip positive on faster storage and will post comparative results next week.