ggml-openvino: Update PERMUTE and CONCAT dynamic dim axis selection, Keep fp32 precision for sum-of-squares by swathiananth · Pull Request #358 · ravi9/llama.cpp

swathiananth · 2026-10-21T21:34:01Z

Overview

Fixes Gemma 3n (E2B and E4B) on OpenVINO GPU, which gave NaN, blank or looping output on prompts of about 700 tokens or more. There are two parts:

fp32 precision for sum-of-squares: a new GPU-only pass keeps x*x → ReduceSum → (scale) → Sqrt in fp32, so it can no longer overflow fp16. The graph structure doesn't change.
Dynamic-dim fix: two bugs in compute_node_dynamic_dims (PERMUTE and CONCAT) marked the wrong axis as dynamic for Gemma 3n. That made the GPU matmul fail to compile.

1. New pass: KeepSumSquareFP32

Files: keep_sum_square_fp32.h, keep_sum_square_fp32.cpp, registered in pipelines/gpu.cpp.

Problem:

The GPU plugin runs fp32 graphs at fp16 inference precision. Gemma 3n's gaussian_topk (first 10 layers) computes a standard deviation over the 8192-wide FFN gate: sqrt(sum((x - mean)^2) / (n-1)).

The sum of squares reaches about 121k, above fp16's maximum of 65,504, so it becomes inf.
The cutoff becomes infinite, ReLU zeroes the whole gate, and the FFN contributes nothing.
Longer prompts bring in higher-energy tokens, which is why it only shows up past a certain length.

What the pass does:

It matches ReduceSum(Multiply(x, x)), where both Multiply inputs are the same tensor.
It marks the square and the ReduceSum with ov::disable_conversion(node, f16).
If the single consumer is a Multiply by a scalar constant (the ggml_scale, e.g. 1/(n-1)), it marks that too. It then marks a following Sqrt.
OpenVINO's ConvertPrecision skips marked nodes, so the GPU plugin runs them in fp32 and adds the Converts at the edges. The Sqrt output is back in fp16 range.
The GPU plugin uses the same mechanism in its own fixes, for example DisableFP16CompForGemma3RMSPattern.

2. ggml-decoder.cpp:

Dynamic-dimension tracking PERMUTE. In ggml, op_params[i] is the destination axis of source dim i.
Uses op_params[dynamic_dim_idx] directly with a bounds check.

CONCAT- First takes a source's dynamic dim. If both sources have one and they disagree, it keeps src[0]'s and logs a warning. It falls back to the axis that grew (concat axis) only if neither source is dynamic.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes. AI was used for analysis and testing.

When the SWA cache size matches the full cache size, identify SWA layers by the number of layers using each mask instead of cache size; the larger group is SWA, and a tie leaves the layers unclassified.

Give SWA KV cache index tensors unique input names (a _swa suffix, like the masks) to prevent them from being merged with the full-attention cache indices.

Two attention masks do not imply SWA: DeepSeek sparse attention uses two full-attention masks with the same name. When cache extents match, classify the larger mask group as SWA only once its mask has fewer keys per row than the other mask. Keep the classification per KV cache buffer so new sequences reuse the same graph, and include swa_layers when checking graph reuse.

Waiting for the window to appear in the mask caused a mid-sequence graph recompile (~9 s for Gemma-4 at -c 1024 after position 512). SWA classification already gives each mask group its own mask, KV write indices, and attention size, which is correct whether or not the mask is a sliding window. Stateful mask rebuild recovers the window from the mask, or the causal mask for full attention. Classify the larger of two mask groups from the first graph and drop the per-KV-buffer bookkeeping.

Add a TODO above the equal-extent fallback to document that the larger mask group is currently assumed to be the SWA group. Since stateless graphs only require the two groups to remain distinct, this heuristic may only be necessary for stateful execution and could potentially be restricted to that path.

Dynamic CPU/GPU graphs previously passed n_seq_active and attention_size(_swa) as separate inputs. Derive these values directly from the KQ mask shape using ShapeOf + Gather inside the model. This allows the GPU plugin to infer the Q reshape and KV slice shapes from shape inference rather than runtime input values. Q's head size also becomes static, enabling the fused SDPA kernel. GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK=1 restores the previous separate-input behavior. The fused SDPA path also requires two supporting changes: - Clamp the GPU mask to -30000. A key block that is entirely -inf for a row produces NaNs in the fused kernel. - Restate the head count and head size after the KV slice. The plugin loses all slice dimensions when the end index is dynamic, causing SDPA to fall back to the reference kernel. Additional graph and quantization improvements: - Skip the identity reshape of rows in SET_ROWS. - Share a single Split for one-element view slices in dynamic GPU graphs (e.g. Gemma-4 per-layer embeddings), replacing one StridedSlice per layer. - Add GGML_OPENVINO_REQUANT_EMBD=native (off by default) to preserve the file's native quantization for token_embd/output instead of requantizing to Q8_0_C. - Include both environment settings in the graph and model cache keys and document them.

The dynamic GPU path reused a Split output for any single-element view by deriving the chunk index from offset / stride. This produced incorrect results for views that were not chunk-aligned or whose remaining dimensions did not preserve the source strides, as seen in test-backend-ops CONT with use_view_slice. Restrict Split reuse to cases where the view exactly matches a split chunk, meaning the offset is aligned to the split stride and the remaining non-unit dimensions preserve the original source strides. Gemma-4 per-layer embedding slices continue to use the shared Split path.

Initialize the newly added `shape_source` and `shape_axis` fields in the `ModelExtraInputInfo` brace initializer used by `add_extra_inputs()`. Without initializers for these fields, GCC emits `-Wmissing-field-initializers`, causing CI to fail with `LLAMA_FATAL_WARNINGS`.

The option identifies embedding and output weights by their tensor names, which are defined by core llama.cpp. Remove it from this PR; preserving the file quantization of these weights will be addressed in a separate change that identifies them by their roles in the graph.

The manual GQA attention path reshaped Q, K, and V to 5-D so that query heads sharing a KV head could broadcast against it. The GPU plugin cannot fuse this graph: it materializes the attention scores and copies the transposed K and V caches for every layer. Instead, stack the query heads for each KV head along the sequence axis (Q [B, H, S, D] -> [B, H_kv, factor * S, D]) and tile the mask accordingly. This keeps the graph 4-D from the MatMuls through the Softmax, allowing the GPU plugin to fuse the attention into its SDPA kernel and fold the K and V transposes into the fused operation. Clamp the mask to -30000, matching the SDPA path, because the fused kernel can return NaN when an entire block is masked with -inf. Models where the K or V head size differs from the Q head size now use the SDPA path, since the manual path assumes one head size.

In stateless GPU mode, bind the entire KV cache as a `[1, ctx, n_heads_kv, head_size]` tensor, with one token per row, instead of binding the first `n_kv` rows as a `[1, 1, ctx, n_heads_kv * head_size]` tensor. Write cache updates with `ScatterUpdate` along the token axis, and read the cache through `VariadicSplit` and `Transpose`. This allows the GPU plugin's `StatelessKVFusion` to replace cache reads and writes with in-place `StatelessKV` nodes, avoiding full-cache copies on every decode step and repeated memory pool allocations as the bound cache size grows. Set `GGML_OPENVINO_DISABLE_KV_4D=1` to retain the previous layout.

With -fa off, llama.cpp stores the V cache transposed, with value elements as rows and tokens along each row. The KV slicing code assumes one token per row and binds the wrong data when n_kv < kv_size. Detect transposed caches with GgmlOvDecoder::is_kvcache_token_rows() and skip slicing for them. For grouped-query attention, avoid repeating K and V for each query head. Stack the query heads for each KV head along the row dimension and let SOFT_MAX apply the scale, mask, and softmax so the GPU plugin can fuse the operation with the MatMuls. Read only the current layer's V-cache rows when layers use different cache sizes. Reject -fa off in stateful mode with a clear error and document that it requires stateless mode.

With `-fa off`, the graph writes the V cache in transposed form, which is not supported by the stateful KV cache. Detect this pattern in `compute_llm_params()` as a `SET_ROWS` into a KV cache through a `[1, n]` reshape. When stateful execution is enabled, log a warning and fall back to stateless execution instead of failing graph conversion. Update the `-fa off` note in `OPENVINO.md` to document this behavior. Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>

With -fa off, the 4-D KV cache layout introduced in ravi9#344 breaks MUL_MAT attention for Phi-3.5 mini and Phi-4 mini, causing oneDNN to fail with "could not construct a memory descriptor using strides". Preserve the [1, 1, ctx, n_heads_kv * head_size] layout when the graph writes a transposed V cache.

Q5_0 weights are requantized to Q4_0, but test-backend-ops builds naive graphs, where requantization is skipped. Q5_0 then reached the normal weight extraction, which had no Q5_0 case, so every Q5_0 MUL_MAT and GET_ROWS op test failed with "Unsupported quantized type: q5_0". Extract Q5_0 as symmetric i8 weights (q - 16) with one f16 scale per 32 weights, the same layout as Q8_0. Models are unchanged: they still requantize Q5_0 to Q4_0.

The NPU GatedDeltaNet kernel writes its output in FP16 and flushes values below 6.1e-5 to zero.

1. Add envs desc into doc 2. Refine the code with removing duplicated env

Reduce host overhead outside `infer()` from 0.5–1.1 ms to 0.14–0.38 ms per token on Panther Lake. - Reuse KV cache tensors and skip redundant port bindings. - Use persistent USM host tensors for small GPU inputs and outputs. - Defer graph-wide consumer lookup until after shape checks. - Return model and compute params by const reference. - Preserve tensor extras when memory, shape and type are unchanged.

Add GGML_OPENVINO_SELF_CONTAINED_RELEASE_MMAP_PAGES as an explicit opt-in for self-contained compiled-model cache hits. Before importing a valid self-contained blob, recursively walk the graph to find unique external GGUF mmap-backed weight buffers and drop their resident pages while preserving the mapping address and tensor pointers. On Linux use page-aligned madvise(MADV_DONTNEED). On Windows, where there is no reliable per-range mapped-file equivalent, call VirtualUnlock for the identified range and trim the current process working set with SetProcessWorkingSetSize(-1, -1). Pages required later can fault back on demand. Keep the behavior disabled by default and document the physical-memory versus commit-memory tradeoff. Profiling reports the submitted GGUF range and model import duration. Validated with ReleaseOV llama-bench/llama-simple build and pp3072+tg512 warm mmap lazy import. With page release enabled, PrefillPeakWS dropped from about 5758 MiB to 3053 MiB while throughput remained comparable; PeakCommit stayed about 3331 MiB.

…eights bank When set, prefill and decode self-contained blobs use one shared NPUW_WEIGHTS_BANK name (ggml-sc-shared-<pfp>-<dfp>) instead of separate per-phase banks. Since both decoders are built from the same model_weights map, NPUW dedups them into a single in-process bank; on warm import the identical (uid,tensor) pairs are content-verified and reused (patched NPUW weights_bank). Cuts resident compiled-weight footprint ~2GB (pp3072 warm: In-Use 7954->5871 MiB, WS 7891->5765) with prefill throughput unchanged.

The shared weights bank (GGML_OPENVINO_NPU_SHARED_BANK) supersedes both: it deduplicates the prefill/decode compiled-weight copy directly, so lazy per-phase import and GGUF-page release no longer reduce steady memory. Remove the two env vars, the self-contained lazy-import branches (defaulting to the eager path that imports both blobs), the release_external_weight_pages helper, and the now-unused ggml_openvino_buffer_is_external / _release_external_pages buffer APIs.

In strict self-contained warm-mmap mode weights come via buffer_from_host_ptr, so every alloc_buffer is graph scratch/KV. The large prefill compute buffer is never touched on the host by the on-device fused OV model, yet the NPU L0 host tensor (create_host_tensor + memset) makes all ~2.4 GiB resident. For buffers >= 1 GiB in this mode, reserve demand-zero pages (no L0, no memset) so they stay non-resident; KV/small buffers keep the L0 host tensor for zero-copy import. pp3072 warm mmap + shared bank: prefill peak WS 5765 -> 3314 MiB (-43%), In-Use delta ~5940 -> ~3232 MiB; prefill 1256.7 -> 1252.9 t/s and decode 28.27 -> 27.96 t/s (both flat). Commit charge unchanged (demand-paged).

The static graph built the sliding-window KQ mask at the full context length, while the windowed layers bind their (smaller) window KV cache. With gemma-4 at -c 4096 -ub 1024 the window cache is 1536 cells, so SDPA shape inference failed ([1,1,1024,4096] mask vs [1,8,1536,256] K/V) and every prompt errored out. With GGML_OPENVINO_NPU_KV_SLICE the prefill mask was meant to use attention_size_swa, but the check matched "swa" in the ggml tensor name, and llama.cpp names both masks "attn_inp_kq_mask"; the windowed layers got the full-attention sliced length instead ([1,1,1024,256] vs [1,8,1024,256]). Identify the sliding-window mask by tensor identity (is_swa_mask) and size it to ctx_per_seq_swa, both in the graph input shape and in the runtime mask padding.

make_dirs() mkdir'ed every path prefix, including the drive root "D:", which fails with EACCES (not EEXIST) on Windows. Any absolute cache path therefore logged "could not create model cache dir ... caching disabled" and silently ran without GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. Skip the drive root.

…red code Mechanical refactor, no behavior change (step 1 of the NPU/MoE architecture plan): * openvino/pipelines/: TranslateSession::apply_transformations() now registers common passes, then the device's passes, then the execution-mode passes (stateful KV, static shapes), selected by a per-graph GraphTraits. The GPU MoE fusions and the GPU eltwise-rank workaround move to pipelines/gpu.cpp; CPU and NPU pipelines are empty placeholders for device lowerings. Pass order is unchanged. * npu/npu_config.*: the NPU (NPUW) compile config built at backend init, the NPUW strip for cacheless graphs and the per-graph static prefill/decode variants (HFA prefill, decode NPUW_UNFOLD_IREQS) move out of ggml-openvino-extra.cpp and utils.cpp. * npu/npu_quant_policy.*: the NPU requantization rules (requant policy, token_embd Q6_K, keep-Q4_0) move out of ggml_openvino_get_requant_type(). Verified identical output before/after on OV-CPU, GPU and NPU (gemma-4 E4B).

KV-cache slicing (bind the KV cache, and size the KQ masks, at the attended length instead of the full context) existed twice: on by default for dynamic graphs (GGML_OPENVINO_DISABLE_KV_SLICE to turn it off) and opt-in for static prefill (GGML_OPENVINO_NPU_KV_SLICE). The static mode is not NPU-specific; it applies to any static graph, including CPU/GPU with GGML_OPENVINO_FORCE_STATIC. Describe it as one mechanism with two modes behind GGML_OPENVINO_KV_SLICE (unset = each mode's default, 0 = off, 1 = on) and two predicates, ggml_openvino_{dynamic,static}_kv_slice_enabled(), used by the KV binding, the static graph shapes, the mask padding and the cache keys. The two old variables remain as deprecated aliases.

…inputs dev_backend_openvino added shape_source/shape_axis to ModelExtraInputInfo; the HFA past-KV and mask initializers still listed four fields, which GCC rejects under -Werror=missing-field-initializers.

…ersion - Convert the static prefill and decode models one after the other, both before either is compiled. The decoders share the weight nodes; converting on two threads crashed intermittently inside libopenvino, and converting decode after the prefill compile gave wrong output on Bonsai-8B. - Compile prefill and decode in parallel only with GGML_OPENVINO_NPU_PARALLEL_COMPILE=1, since it raises the host memory peak. - Unfold NPUW function calls on the decode graph only with GGML_OPENVINO_NPU_DECODE_UNFOLD=1. The unfold default gave wrong output on gemma-4-E4B and slowed decode on several small models. - Stream NPU weight requantization only with GGML_OPENVINO_REDUCE_COMPILE_MEM or GGML_OPENVINO_MEMORY_OPTIMIZE, as on the other devices. - Fix a -Werror=missing-field-initializers build error in utils.cpp. - Declare ggml_openvino_get_npu_requant_type() only in npu/npu_quant_policy.h. - Docs: document the new switches, deprecate stateful execution, show stateless-only results in the validated models table, and update the validation system.

…raph

…nvert. Prevents perf degrade.

swathiananth · 2026-10-30T00:29:57Z

Date: 2026-10-29

Setup

Host: DUT6330BMGFRD (10.36.25.33), Windows 11, Intel Arc Pro B70 32 GB
OpenVINO: 2026.4 (C:\sananth2\openvino_2026.4). Builds use VS 2026, Release, GGML_OPENVINO=ON
Models: C:\sananth2\Models\gemma-3n-E2B-it-Q4_K_M.gguf, gemma-3n-E4B-it-Q4_K_M.gguf
Prompt: C:\sananth2\g3n_exp\prompt_1024.txt (979 tokens)

Command:

export GGML_OPENVINO_DEVICE=GPU
export GGML_OPENVINO_STATEFUL_EXECUTION=1
./llama-completion -m <model> -f prompt_1024.txt -c 1382 -n 128 -st \
  -ngl -1 -no-cnv --ignore-eos

llama.cpp rounds the context up to n_ctx = 1536.

Test Result:

Model	rc	Load	Prompt eval (979 tok)	Generation (127 runs)	Output
gemma-3n-E2B-it-Q4_K_M	0	24.1 s	1685.65 ms, 580.78 tok/s	3695.18 ms, 34.37 tok/s	Coherent ("Briefing to the Alder Creek Town Council…")
gemma-3n-E4B-it-Q4_K_M	0	28.9 s	1803.63 ms, 542.80 tok/s	4331.75 ms, 29.32 tok/s	Coherent ("Alder Creek Library Improvement Plan…")

cavusmustafa · 2026-10-30T21:02:53Z

LGTM, needs rebase

…_3n_E2B_E4B_Qwen3_VL_fixes To get my branch upto date.

swathiananth · 2026-10-30T22:25:54Z

LGTM, needs rebase

Have merged with the remote.

swathiananth requested review from cavusmustafa and wine99 as code owners October 21, 2026 21:34

swathiananth marked this pull request as draft October 21, 2026 21:34

github-actions Bot added documentation Improvements or additions to documentation OpenVINO ggml labels Oct 21, 2026

ravi9 force-pushed the dev_backend_openvino branch from c5cd082 to 947be59 Compare October 24, 2026 06:11

osabnis and others added 23 commits October 26, 2026 11:16

ggml-openvino: fix SWA mask detection

61836a7

When the SWA cache size matches the full cache size, identify SWA layers by the number of layers using each mask instead of cache size; the larger group is SWA, and a tie leaves the layers unclassified.

ggml-openvino: fix SWA KV cache write indices

34f4835

Give SWA KV cache index tensors unique input names (a _swa suffix, like the masks) to prevent them from being merged with the full-attention cache indices.

make embedding to have dynamic shape for OV GPU

b68bb26

ggml-openvino: make the 4-D stateless KV cache opt-in

8f9b64e

add support for Q5_0 data type in OV GPU backend

e160620

resolve CI test error

f7a9350

ggml-openvino: Optimize GDN output pre-scale for the NPU static path

99afbf5

The NPU GatedDeltaNet kernel writes its output in FP16 and flushes values below 6.1e-5 to zero.

Refine the code as review comments

74636e7

1. Add envs desc into doc 2. Refine the code with removing duplicated env

ggml-openvino: Add npuw dump envs for profiling

a56a337

zhaixuejun1993 and others added 11 commits October 26, 2026 11:16

perf improvement success

1bb99e8

ggml-openvino: initialize all ModelExtraInputInfo fields for the HFA …

db3dc95

…inputs dev_backend_openvino added shape_source/shape_axis to ModelExtraInputInfo; the HFA past-KV and mask initializers still listed four fields, which GCC rejects under -Werror=missing-field-initializers.

ravi9 force-pushed the dev_backend_openvino branch from 674af4a to 9e01c2f Compare October 26, 2026 21:41

Update fix to just maintain fp32 precision instead of modifying the g…

84bb11d

…raph

swathiananth closed this Oct 26, 2026

swathiananth force-pushed the Gemma_3n_E2B_E4B_Qwen3_VL_fixes branch from 865a7a8 to 9e01c2f Compare October 26, 2026 22:48

swathiananth reopened this Oct 26, 2026

swathiananth changed the title ~~ggml-openvino: Update CONCAT dynamic dim axis selection, fix fp16 overflow for L2Norm, change pad to handle rank < 4~~ ggml-openvino: Update PERMUTE and CONCAT dynamic dim axis selection, Keep fp32 precision for sum-of-squares Oct 26, 2026

Limit the fp32 precision spreading in the graph by fencing it with Co…

131fa2f

…nvert. Prevents perf degrade.

ravi9 force-pushed the dev_backend_openvino branch 2 times, most recently from be796d8 to 25e3135 Compare October 29, 2026 17:37

swathiananth added 4 commits October 29, 2026 23:39

Merge branch 'dev_backend_openvino' into Gemma_3n_E2B_E4B_Qwen3_VL_fixes

af51388

Update OPENVINO.md

1a253e0

Update flash_attn_ext.cpp

e829190

Update utils.cpp

405cf9c

swathiananth marked this pull request as ready for review October 30, 2026 00:30

Merge remote-tracking branch 'origin/dev_backend_openvino' into Gemma…

f65c1a1

…_3n_E2B_E4B_Qwen3_VL_fixes To get my branch upto date.

cavusmustafa merged commit c63b77e into ravi9:dev_backend_openvino Nov 2, 2026
4 of 5 checks passed

Navigation Menu

Repository navigation

ggml-openvino: Update PERMUTE and CONCAT dynamic dim axis selection, Keep fp32 precision for sum-of-squares - #358

ggml-openvino: Update PERMUTE and CONCAT dynamic dim axis selection, Keep fp32 precision for sum-of-squares#358
cavusmustafa merged 59 commits into
ravi9:dev_backend_openvinofrom
swathiananth:Gemma_3n_E2B_E4B_Qwen3_VL_fixes

swathiananth commented Oct 21, 2026 •

edited

Loading

Uh oh!

swathiananth commented Oct 30, 2026 •

edited

Loading

Uh oh!

cavusmustafa commented Oct 30, 2026

Uh oh!

swathiananth commented Oct 30, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Conversation

swathiananth commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

1. New pass: KeepSumSquareFP32

Problem:

What the pass does:

2. ggml-decoder.cpp:

Requirements

Uh oh!

swathiananth commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Setup

Test Result:

Uh oh!

cavusmustafa commented Oct 30, 2026

Uh oh!

swathiananth commented Oct 30, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

swathiananth commented Oct 21, 2026 •

edited

Loading

swathiananth commented Oct 30, 2026 •

edited

Loading