[osabnis]

Overview

Gemma-2 caps its attention scores as softcap * tanh(scores / softcap) before the softmax. The OpenVINO backend did not support this, so Gemma-2 gave wrong output in stateless mode with both -fa on and -fa off.

• FLASH_ATTN_EXT (-fa on): applies the soft-capping on the manual attention path, since OpenVINO's SDPA has no soft-capping. supports_op now accepts soft-capped attention when Q, K and V share one head size.
• Attention matching (-fa off): the decoder looks through the SCALE, TANH, SCALE chain between the KQ product and SOFT_MAX.
• Sliding-window layers: one helper now finds the K cache of an attention for both the sliding-window pre-pass and the main pass. The pre-pass had its own copy, which did not see through the chain, so Gemma-2's sliding-window layers were treated as full-attention layers. This is harmless while both KV caches have the same size, and gives wrong output once the context exceeds n_swa + n_ubatch (4096 + 512 tokens).
• PERMUTE: for the transposed V cache of -fa off, the attended tokens are taken from the last axis.
• 4-D KV cache (GGML_OPENVINO_ENABLE_KV_4D): soft-capped attention keeps the flat KV layout, as the GPU plugin fails in its oneDNN path with the 4-D layout ("could not construct a memory descriptor using strides"). Same approach as the existing exclusion for -fa off.

Additional information

Tested on Arc B580 with an OpenVINO master build from 2026-10-15, Gemma-2 2B Q4_K_M, before and after this change on the current dev_backend_openvino.

Accuracy (stateless)

KL divergence against the CPU backend:

	ctx 8192	ctx 1024, one-token decode	Output at ctx 8192
-fa on, before	fails	—	garbled
-fa on, after	0.0345	0.0384	correct
-fa off, before	fails	—	garbled
-fa off, after	0.0343	0.0375	correct

With GGML_OPENVINO_ENABLE_KV_4D=1 the results are the same (0.0345 and 0.0343 at ctx 8192); before this change, -fa on failed with it.

Decode performance (stateless, after)

tg128 tok/s at depth 0 / 4096: -fa on 110.6 / 66.0, -fa off 112.7 / 74.8.

Stateful

Stateful execution works with this change but is not optimized: soft-capped attention cannot fuse into SDPA, so it runs as separate matmuls (-fa on: 68.4 / 59.8 tok/s at depth 0 / 4096). With -fa off, stateful execution needs #361; without it, the context falls back to stateless as before.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - AI assistance was used for analysis and running tests.

[osabnis]

Update:

• Panther Lake iGPU, OpenVINO master build, GGML_OPENVINO_ENABLE_KV_4D=1.
• Gemma-2 2B Q4_K_M. Base = dev_backend_openvino 674af4a (the tip on 10-26), PR = base + ggml-openvino: support gemma-2 soft-capped attention #362.
• KLD against the CPU backend, ctx 1024, one-token decode. llama-bench -r 5.
• Prompt "The capital of France is", -c 8192.

Mode	Base	With #362
stateless -fa on	wrong answer, KLD 10.3, tg128 13.4 tok/s	"a) Paris", KLD 0.0384, tg128 45.9 / 26.2 tok/s (d0 / d4096), pp512 3252 / 2014
stateless -fa off	wrong answer, KLD fails	"a) Paris", KLD 0.0375, tg128 46.4 / 29.6, pp512 3017 / 1539
stateful -fa on	fails (set_rows.cpp:62 shape check)	"a) Paris", KLD 0.0375, tg128 46.9 / 33.1, pp512 2431 / 1190
stateful -fa off	wrong answer, KLD fails (falls back to stateless)	"a) Paris", KLD 0.0375, tg128 39.2 / 11.8 (falls back to stateless)
stateful -fa off + #361	-	"a) Paris", KLD 0.0375, tg128 44.4 / 36.8, runs stateful

Same KLD as on the B580 (0.0384 / 0.0375).