[osabnis]

Overview

Depends on #365
With -fa off, llama.cpp writes the V cache transposed. Stateful execution did not support this layout, so these contexts fell back to stateless execution (ravi9/llama.cpp#346), where the KV cache stays in host memory and is read every token.

This PR keeps the V cache token-major in stateful mode, like K, and removes the fallback.

• SET_ROWS: appends the V values directly to the state in token order.
• PERMUTE: reads V as [1, n_heads_kv, kv, head_size] and marks it so MUL_MAT can consume it without another transpose.
• MUL_MAT: removes an unnecessary reshape in grouped attention, so OpenVINO's SDPA fusion matches the -fa off pattern. Without the fusion, stateful execution was slower than the stateless fallback at short contexts.
• Gemma-4: moves the V head-size slice after the transpose so it reads the padded state in place. This avoids copying the growing KV cache and recompiling that copy for every KV length.
• PAD: supports the rank-3 tensors used by stateful execution.
• Removes the "-fa off: stateless only" note from docs/backend/OPENVINO.md.

Additional information

Tested on Panther Lake (Arc B390) and Arc B580 against OpenVINO master. Measured on 38d09d2 and rebased onto the current dev_backend_openvino; the B580 accuracy check below was repeated after the rebase, with the same results.

Decode performance

Stateful -fa off, tg128 tok/s on Panther Lake, before → after:

Model	Depth 0	Depth 4096
Llama-3.2 1B	87.96 → 104.40	30.28 → 80.25
SmolLM2 1.7B	46.49 → 79.25	7.26 → 45.81
Phi-4-mini	27.92 → 33.92	9.15 → 27.15
Phi-3.5-mini	22.15 → 35.50	3.79 → 23.34
MiniCPM-o 2.6	19.79 → 21.21	11.56 → 19.19
Qwen3.5 4B	23.31 → 26.65	—*
Gemma-4 E2B	—†	—†

* llama-bench's depth-4096 run rewinds the recurrent state of Qwen3.5, which stateful execution cannot do; before this change it ran only because the context fell back to stateless execution.
† llama-bench fails for Gemma-4 with -fa off before and after this change, also in stateless mode. Its accuracy is covered below.

With this change, stateful -fa off decodes at the speed of stateful -fa on (Llama-3.2 1B: 104.40 vs 105.08 tok/s at depth 0). The other three configurations (-fa on stateful/stateless and -fa off stateless) are unchanged: decode within -1.3% to +0.9%.

Accuracy

KL divergence against the CPU backend (ctx 1024, one-token decode) for stateful -fa off matches stateless -fa off within 0.2% on Panther Lake for all six models with a reference: for example, Llama-3.2 1B 0.0406 vs 0.0406, Phi-4-mini 0.0356 vs 0.0357, Gemma-4 0.3634 vs 0.3635.

On B580 the same holds for Llama-3.2 1B, Phi-3.5, Phi-4 and Gemma-4, with -b 1024 and with one-token decode (-b 1).

Limitation

If a model pads V rows across layers and has more than one KV head per layer, the V slice still requires a reshape and copies the cache. None of the tested models hit this case; Gemma-4 has one KV head.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - AI assistance was used for analysis and running tests.