Summary
Running any DeepSeek V4 PRO GGUF on the CPU backend aborts almost immediately at prefill layer 1 with:
ds4: grouped Q8_0 tensor has an unexpected layout
Flash works fine on the same CPU build; only PRO is affected. The root cause is a hardcoded Flash-specific constant in the CPU attention-output projection.
Environment
- CPU backend (
make cpu), x86-64 Linux (Ubuntu 24.04, Xeon w9-3595X, 503 GB RAM)
- Model:
DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix.gguf (PRO Q2)
- ds4 @
main
Reproduce
make cpu
./ds4 -m DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix.gguf \
--cpu -c 2048 -n 48 --nothink -p "hello"
# ds4: prefill layer 1/61ds4: grouped Q8_0 tensor has an unexpected layout (exit 1)Root cause
layer_grouped_out_one(), layer_grouped_out_one_decode_scratch(), and layer_grouped_out_batch() (ds4.c) hardcode the attention-output group count:
const uint32_t n_groups = 8; // Flash value
PRO's shape profile sets n_out_group = 16 (Flash = 8). With the literal 8, PRO computes
group_dim = n_head_dim * (n_head / n_groups) = 512 * (128 / 8) = 8192, but PRO's attn_output_a
tensor has dim[0] = 512 * (128 / 16) = 4096, so the w->dim[0] != group_dim check in
matmul_q8_0_grouped_batch() / matvec_q8_0_grouped_rows() fails and ds4_die() fires.
The engine already exposes the correct value as the DS4_N_OUT_GROUP macro (= g_ds4_shape.n_out_group); the CPU path just doesn't use it at these three sites.
Fix
Replace the three hardcoded n_groups = 8 with n_groups = DS4_N_OUT_GROUP. rank = 1024 is shared across profiles, so no other change is needed. Flash is unaffected (macro == 8); PRO then prefills all 61 layers and produces coherent output on CPU (~0.95 t/s on this host). PR incoming.
Impact
Minor — the CPU path is the diagnostics/reference backend — but this also blocks using it as a correctness oracle for PRO. GPU/Metal backends are unaffected.
Summary
Running any DeepSeek V4 PRO GGUF on the CPU backend aborts almost immediately at prefill layer 1 with:
Flash works fine on the same CPU build; only PRO is affected. The root cause is a hardcoded Flash-specific constant in the CPU attention-output projection.
Environment
make cpu), x86-64 Linux (Ubuntu 24.04, Xeon w9-3595X, 503 GB RAM)DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix.gguf(PRO Q2)mainReproduce
make cpu ./ds4 -m DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix.gguf \ --cpu -c 2048 -n 48 --nothink -p "hello" # ds4: prefill layer 1/61ds4: grouped Q8_0 tensor has an unexpected layout (exit 1)Root cause
layer_grouped_out_one(),layer_grouped_out_one_decode_scratch(), andlayer_grouped_out_batch()(ds4.c) hardcode the attention-output group count:PRO's shape profile sets
n_out_group = 16(Flash = 8). With the literal8, PRO computesgroup_dim = n_head_dim * (n_head / n_groups) = 512 * (128 / 8) = 8192, but PRO'sattn_output_atensor has
dim[0] = 512 * (128 / 16) = 4096, so thew->dim[0] != group_dimcheck inmatmul_q8_0_grouped_batch()/matvec_q8_0_grouped_rows()fails andds4_die()fires.The engine already exposes the correct value as the
DS4_N_OUT_GROUPmacro (= g_ds4_shape.n_out_group); the CPU path just doesn't use it at these three sites.Fix
Replace the three hardcoded
n_groups = 8withn_groups = DS4_N_OUT_GROUP.rank = 1024is shared across profiles, so no other change is needed. Flash is unaffected (macro == 8); PRO then prefills all 61 layers and produces coherent output on CPU (~0.95 t/s on this host). PR incoming.Impact
Minor — the CPU path is the diagnostics/reference backend — but this also blocks using it as a correctness oracle for PRO. GPU/Metal backends are unaffected.