CUDA ssd-streaming: expert cache thrashes on mixed-quant GGUFs (decode ~0.9 t/s); prefill drops the whole cache every request

[iCreil]

Environment

• GPU: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation 96GB (sm_120)
• Driver 595.71.05, CUDA runtime 13.2, toolkit 13.1 (make cuda-generic)
• Host: EPYC 9124, 180 GiB RAM (VM, GPU passthrough), Ubuntu 26.04
• ds4 commit 80ebbc3
• Model: official DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf (q2-q4 mixed, 91GB)

Command:

DS4_CUDA_NO_DIRECT_IO=1 DS4_CUDA_KEEP_MODEL_PAGES=1 ./ds4-server \
  -m <mixed.gguf> --ctx 32768 --host 0.0.0.0 --prefill-chunk 2048 \
  --ssd-streaming --ssd-streaming-cache-experts 4500 \
  --kv-disk-dir /data/ds4-kv --kv-disk-space-mb 16384

Symptom

Decode is stuck at ~0.9 t/s (never ramps up) and stderr floods with thousands of
alternating cap notices, two sizes flip-flopping forever:

ds4: CUDA streaming expert cache capped from 10619 to 4973 experts (available 81.57 GiB, reserve 16.00 GiB, 13.50 MiB/expert)
ds4: CUDA streaming expert cache capped from 10619 to 9947 experts (available 81.57 GiB, reserve 16.00 GiB, 6.75 MiB/expert)
ds4: CUDA streaming expert cache capped from 10619 to 4973 experts (available 81.57 GiB, reserve 16.00 GiB, 13.50 MiB/expert)
... (repeats for the whole run)

A 200-word completion does not finish within 400s. The same setup with the uniform
q2-imatrix or q4-imatrix GGUFs works fine.

Root cause 1 — single-size expert cache released at every size transition

ds4_cuda.cu keeps ONE global g_stream_expert_cache keyed on the last-seen
(gate_expert_bytes, down_expert_bytes):

• cuda_stream_expert_cache_note_size() resets the runtime cap whenever the size
  differs from the previous call;
• cuda_stream_expert_cache_prepare() calls release_all() when dims differ from
  the cached ones.

The mixed GGUF interleaves layers with 6.75 MiB and 13.50 MiB experts. The slab/bypass
logic in ds4.c (weights_streaming_layer_experts_uniform) covers the graph path, but
the CUDA cache layer still receives both sizes (e.g. via
ds4_gpu_stream_expert_cache_budget_for_expert_size and the seed/compact-load paths),
so the cache is torn down and rebuilt at every q2↔q4 layer transition — it never
accumulates a single hit.

Root cause 2 — prefill drops the whole resident cache on every request

ds4_gpu_stream_expert_cache_prepare_selected_batch →
cuda_stream_selected_cache_begin_compact_load(..., allow_global_cache=0) executes:

if (!allow_global_cache) {
    cuda_stream_expert_cache_release_all();
}

unconditionally — i.e. every prompt releases the entire warm expert cache, even for
a 20-token chat prompt whose staging buffers are ~100 MB. Every request then re-warms
the cache from zero during decode (visible as a chunk-rate ramp 1.7 → 11 t/s on every
single request). This affects uniform models too, not just mixed ones.

Fix (see PR)

1. Expert caches per size class (small fixed array keyed by (gate,down) bytes) —
   removes the cross-class teardown entirely;
2. begin_compact_load: try the staging allocation first, release the expert cache
   only if VRAM is actually short (then retry once);
3. Let the prefill batch path read the global cache (hits become device-to-device
   copies) with an append-but-never-evict policy, so a long prompt cannot cycle out
   the decode working set.

Results (same GPU, same mixed GGUF)

build	200-word completion
stock 80ebbc3	does not finish in 400s (~0.9 t/s, log flood)
patched, --ssd-streaming-cache-experts 7500, DS4_CUDA_STREAMING_EXPERT_CACHE_RESERVE_GB=8	~12.6 t/s end-to-end, DS4_CUDA_STREAMING_EXPERT_CACHE_VERBOSE shows 97% hit rate on the 2nd request, no cap-flood
patched, --ssd-streaming-cache-experts 9472 (= all uniform-class experts resident, 62 GiB), reserve 8	~15 t/s end-to-end after convergence

Caveat found while tuning: the per-class budget can overcommit — a count budget of N
lets EVERY class try N slots, and the lazily-growing model-range arena (q8_0 chunks,
~1.8 GiB each) can then fail mid-decode (cuda decode failed). A shared byte budget
across classes (or reserving the arena's projected size upfront) would make sizing
safer; for now the practical rule on 96GB with this GGUF is: budget = uniform-class
expert count (9472), reserve ≥ 8 GB.

Uniform q2/q4 GGUFs behave as before (q2 fully-resident path untouched).

Remaining ceiling on mixed is the per-token device-to-device compact-buffer copies
(~2 GB/token even at 97% hit rate) — that seems related to the direction discussed in
#491 (serving experts directly from cache slots), out of scope here.