Repository navigation
ggml-openvino: Drop leading pad entries for missing dimensions - #360
Merged
cavusmustafa merged 55 commits intoOct 27, 2026
Merged
cavusmustafa merged 55 commits into
cavusmustafa merged 55 commits into
Conversation
swathiananth
changed the title
swathiananth
changed the title
swathiananth
changed the title
When the SWA cache size matches the full cache size, identify SWA layers by the number of layers using each mask instead of cache size; the larger group is SWA, and a tie leaves the layers unclassified.
Give SWA KV cache index tensors unique input names (a _swa suffix, like the masks) to prevent them from being merged with the full-attention cache indices.
Two attention masks do not imply SWA: DeepSeek sparse attention uses two full-attention masks with the same name. When cache extents match, classify the larger mask group as SWA only once its mask has fewer keys per row than the other mask. Keep the classification per KV cache buffer so new sequences reuse the same graph, and include swa_layers when checking graph reuse.
Waiting for the window to appear in the mask caused a mid-sequence graph recompile (~9 s for Gemma-4 at -c 1024 after position 512). SWA classification already gives each mask group its own mask, KV write indices, and attention size, which is correct whether or not the mask is a sliding window. Stateful mask rebuild recovers the window from the mask, or the causal mask for full attention. Classify the larger of two mask groups from the first graph and drop the per-KV-buffer bookkeeping.
Add a TODO above the equal-extent fallback to document that the larger mask group is currently assumed to be the SWA group. Since stateless graphs only require the two groups to remain distinct, this heuristic may only be necessary for stateful execution and could potentially be restricted to that path.
Dynamic CPU/GPU graphs previously passed n_seq_active and attention_size(_swa) as separate inputs. Derive these values directly from the KQ mask shape using ShapeOf + Gather inside the model. This allows the GPU plugin to infer the Q reshape and KV slice shapes from shape inference rather than runtime input values. Q's head size also becomes static, enabling the fused SDPA kernel. GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK=1 restores the previous separate-input behavior. The fused SDPA path also requires two supporting changes: - Clamp the GPU mask to -30000. A key block that is entirely -inf for a row produces NaNs in the fused kernel. - Restate the head count and head size after the KV slice. The plugin loses all slice dimensions when the end index is dynamic, causing SDPA to fall back to the reference kernel. Additional graph and quantization improvements: - Skip the identity reshape of rows in SET_ROWS. - Share a single Split for one-element view slices in dynamic GPU graphs (e.g. Gemma-4 per-layer embeddings), replacing one StridedSlice per layer. - Add GGML_OPENVINO_REQUANT_EMBD=native (off by default) to preserve the file's native quantization for token_embd/output instead of requantizing to Q8_0_C. - Include both environment settings in the graph and model cache keys and document them.
The dynamic GPU path reused a Split output for any single-element view by deriving the chunk index from offset / stride. This produced incorrect results for views that were not chunk-aligned or whose remaining dimensions did not preserve the source strides, as seen in test-backend-ops CONT with use_view_slice. Restrict Split reuse to cases where the view exactly matches a split chunk, meaning the offset is aligned to the split stride and the remaining non-unit dimensions preserve the original source strides. Gemma-4 per-layer embedding slices continue to use the shared Split path.
Initialize the newly added `shape_source` and `shape_axis` fields in the `ModelExtraInputInfo` brace initializer used by `add_extra_inputs()`. Without initializers for these fields, GCC emits `-Wmissing-field-initializers`, causing CI to fail with `LLAMA_FATAL_WARNINGS`.
The option identifies embedding and output weights by their tensor names, which are defined by core llama.cpp. Remove it from this PR; preserving the file quantization of these weights will be addressed in a separate change that identifies them by their roles in the graph.
The manual GQA attention path reshaped Q, K, and V to 5-D so that query heads sharing a KV head could broadcast against it. The GPU plugin cannot fuse this graph: it materializes the attention scores and copies the transposed K and V caches for every layer. Instead, stack the query heads for each KV head along the sequence axis (Q [B, H, S, D] -> [B, H_kv, factor * S, D]) and tile the mask accordingly. This keeps the graph 4-D from the MatMuls through the Softmax, allowing the GPU plugin to fuse the attention into its SDPA kernel and fold the K and V transposes into the fused operation. Clamp the mask to -30000, matching the SDPA path, because the fused kernel can return NaN when an entire block is masked with -inf. Models where the K or V head size differs from the Q head size now use the SDPA path, since the manual path assumes one head size.
In stateless GPU mode, bind the entire KV cache as a `[1, ctx, n_heads_kv, head_size]` tensor, with one token per row, instead of binding the first `n_kv` rows as a `[1, 1, ctx, n_heads_kv * head_size]` tensor. Write cache updates with `ScatterUpdate` along the token axis, and read the cache through `VariadicSplit` and `Transpose`. This allows the GPU plugin's `StatelessKVFusion` to replace cache reads and writes with in-place `StatelessKV` nodes, avoiding full-cache copies on every decode step and repeated memory pool allocations as the bound cache size grows. Set `GGML_OPENVINO_DISABLE_KV_4D=1` to retain the previous layout.
With -fa off, llama.cpp stores the V cache transposed, with value elements as rows and tokens along each row. The KV slicing code assumes one token per row and binds the wrong data when n_kv < kv_size. Detect transposed caches with GgmlOvDecoder::is_kvcache_token_rows() and skip slicing for them. For grouped-query attention, avoid repeating K and V for each query head. Stack the query heads for each KV head along the row dimension and let SOFT_MAX apply the scale, mask, and softmax so the GPU plugin can fuse the operation with the MatMuls. Read only the current layer's V-cache rows when layers use different cache sizes. Reject -fa off in stateful mode with a clear error and document that it requires stateless mode.
With `-fa off`, the graph writes the V cache in transposed form, which is not supported by the stateful KV cache. Detect this pattern in `compute_llm_params()` as a `SET_ROWS` into a KV cache through a `[1, n]` reshape. When stateful execution is enabled, log a warning and fall back to stateless execution instead of failing graph conversion. Update the `-fa off` note in `OPENVINO.md` to document this behavior. Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
With -fa off, the 4-D KV cache layout introduced in ravi9#344 breaks MUL_MAT attention for Phi-3.5 mini and Phi-4 mini, causing oneDNN to fail with "could not construct a memory descriptor using strides". Preserve the [1, 1, ctx, n_heads_kv * head_size] layout when the graph writes a transposed V cache.
Q5_0 weights are requantized to Q4_0, but test-backend-ops builds naive graphs, where requantization is skipped. Q5_0 then reached the normal weight extraction, which had no Q5_0 case, so every Q5_0 MUL_MAT and GET_ROWS op test failed with "Unsupported quantized type: q5_0". Extract Q5_0 as symmetric i8 weights (q - 16) with one f16 scale per 32 weights, the same layout as Q8_0. Models are unchanged: they still requantize Q5_0 to Q4_0.
The NPU GatedDeltaNet kernel writes its output in FP16 and flushes values below 6.1e-5 to zero.
1. Add envs desc into doc 2. Refine the code with removing duplicated env
Reduce host overhead outside `infer()` from 0.5–1.1 ms to 0.14–0.38 ms per token on Panther Lake. - Reuse KV cache tensors and skip redundant port bindings. - Use persistent USM host tensors for small GPU inputs and outputs. - Defer graph-wide consumer lookup until after shape checks. - Return model and compute params by const reference. - Preserve tensor extras when memory, shape and type are unchanged.
Extend the GGML_OPENVINO_RELEASE_WEIGHTS safety check to the static graph cache-miss path. Once host weight buffers have been released, any later compile would read invalid/zeroed host pages and bake the wrong constants into a new OpenVINO model. The dynamic path already rejected this situation. Static execution can still miss the decoder cache when graph shapes or model parameters change, so it needs the same guard before erasing cached infer requests and rebuilding prefill/decode models. This is a correctness and diagnosability change rather than a throughput optimization. It preserves the memory-saving release-weights mode for stable graph shapes, but turns unsupported recompilation into an explicit abort instead of silently producing a corrupted compiled model.
Add GGML_OPENVINO_NPU_KEEP_Q4_0 as an opt-in escape hatch for NPU weight handling. When enabled for native Q4_0 tensors, the NPU requant path returns nullopt and keeps the original Q4_0 extraction instead of regrouping the tensor through the selected NPU requant policy. The motivation is to avoid unnecessary work for weights that are already stored as 4-bit values. The default NPU policy maps quantized weights to layouts such as Q4_0_128, which can reduce scale metadata but requires round-tripping Q4_0 through the requantization path. Keeping Q4_0 native can save that conversion work and preserve the original group-32 packing. Keep the option experimental and disabled by default. Current NPU driver/plugin stacks can compile the resulting group-32 graph, but inference may hang or fail with ZE_RESULT_ERROR_DEVICE_LOST. The default behavior therefore continues to use the safer NPU requant policy; this flag is mainly useful for debugging driver behavior, measuring the true cost of regrouping, and testing future NPU stacks. Trade-off: native Q4_0 avoids regrouping overhead and keeps the original quantization blocks, but it gives up the more regular group-128 layout expected by the default NPU compressed-weight path and may be unstable on current hardware/software combinations.
Compile the static prefill and decode models with separate OpenVINO config maps so the decode graph can opt into NPUW_UNFOLD_IREQS without forcing the same lowering on prefill. The build_static_model helper now receives the config map explicitly, and the decode path gets a copy of the base config with NPUW_UNFOLD_IREQS=YES when the user has not already set it. The performance target is NPU token generation. Decode executes a long stream of single-token infers, where folded NPUW function-call dispatch overhead becomes visible in every token. Unfolding the NPUW calls into separate infer requests removes that repeated dispatch cost and measured about +20% token-generation throughput on the tested stack, matching the behavior seen in OpenVINO GenAI. Prefill keeps the original folded config. Prompt processing is a batched workload dominated by larger matmuls, and the folded form is faster there while also using less memory. Splitting the config lets prefill and decode use the lowering that matches their workload shape instead of treating NPUW_UNFOLD_IREQS as a single global compromise. Trade-off: unfolding can increase memory and request-management overhead, so the automatic default is limited to the NPU decode model. If a user explicitly provides NPUW_UNFOLD_IREQS through the named env alias or GGML_OPENVINO_NPU_CONFIG, that value is respected because the decode override is only applied when the property is absent.
Collect the remaining small cleanups from the NPU tuning series. This reorders the NPU KV-slice documentation and environment-variable cache entry next to the other NPU opt-in toggles, adds a short header comment for the centralized NPU helper declarations, and makes the static prefill attention-size assignment in the decoder a little more explicit. There is no intended behavior change in this commit. The purpose is to make the final source layout match the split commits that introduced the NPU tuning knobs, so future readers can find the related flags and helpers in one place without mixing that mechanical cleanup into the functional performance commits.
Move the default NPU_COMPILER_TYPE=DRIVER setting into the NPU compile_config initializer so the default NPU plugin options are declared in one place. Route GGML_OPENVINO_NPU_COMPILE_CONFIG and GGML_OPENVINO_NPU_COMPILER_TYPE through the same set_compile_option_from_env helper used by the other named NPU compile option aliases. This keeps the legacy NPU_COMPILE_CONFIG mapping to NPU_COMPILATION_MODE_PARAMS intact while making all optional env-to-compile_config overrides follow the same non-empty-value rule. GGML_OPENVINO_NPU_CONFIG is still parsed last, so explicit comma-separated KEY=VALUE overrides continue to have the highest precedence over both defaults and named aliases.
Default the streaming requantization path on for NPU so the compile-time F32 dequant transient (~1-2 GB for token_embd) is capped without an env flag. Output is byte-identical to full materialization; GGML_OPENVINO_REDUCE_COMPILE_MEM / MEMORY_OPTIMIZE still override (set to 0 to force off). Kept the g_nonov_weight_cache opt-in to avoid holding token_embd resident (which regressed decode steady memory). pp1024_d0: PrefillPeakPrivWS 5974->4664 MiB (-22%), decode steady and pp/tg throughput unchanged.
Add an opt-in embedded-weight cache mode for the static NPU backend. GGML_OPENVINO_SELF_CONTAINED_BLOB=1 exports separate prefill and decode compiled models with ENABLE_WEIGHTLESS=false, uses isolated NPUW weight banks, validates raw GGUF weight manifests, binds imported ports by deterministic frontend order, and serializes the large NPUW compile/export/import operations. Preserve raw model bytes during cold cache construction by requantizing into separately owned OV tensors instead of rewriting the backend model buffer. This keeps later graph fingerprints and manifests stable when llama-bench creates independent prompt and generation contexts. Add strict GGML_OPENVINO_SELF_CONTAINED_MMAP=1 warm-import mode. The backend exposes a non-owning buffer_from_host_ptr wrapper over the GGUF mmap, uses raw tensor allocation sizes, avoids allocating/copying the private OpenVINO model-weight buffer, and stages the few mmap-backed runtime constants into small importable NPU host tensors. Invalid or missing cache entries fail clearly instead of trying to requantize into read-only mapped storage. The normal path and existing weightless MODEL_PTR cache remain unchanged when the new flags are unset. Verified with phi-4-mini: chunk512 pp3072/tg512 cold 1123.10/27.60 t/s and warm mmap 1124.72/27.16 t/s; warm mmap removes roughly 3.9 GiB peak commit in the measured prefill workload. Self-contained blobs consume about 8.34 GiB for the separate pp and tg contexts.
Add GGML_OPENVINO_SELF_CONTAINED_LAZY_IMPORT to make the low-working-set strict mmap path opt in instead of changing the default warm mmap behavior. When the flag is set with SELF_CONTAINED_BLOB=1 and SELF_CONTAINED_MMAP=1, the static NPU path imports only the current phase blob: prefill imports the prefill blob, decode imports the decode blob later. This avoids the pair-shaped shared compiled-model cache for the lazy path and caches only the current phase infer request. The regular warm mmap path remains eager and continues to prepare both prefill and decode compiled models. Non-mmap self-contained builds remain serial, and the regular non-self-contained path keeps the existing parallel build behavior. Also register the new environment variable, document the memory/latency tradeoff, and add profiling-only import timing around model-cache imports. Validated with ReleaseOV build: cmake --build build/ReleaseOV --target llama-bench llama-simple --config Release. Lazy warm mmap run measured PrefillPeakWS around 5759 MiB versus the eager warm mmap path around 7897 MiB.
Add GGML_OPENVINO_SELF_CONTAINED_RELEASE_MMAP_PAGES as an explicit opt-in for self-contained compiled-model cache hits. Before importing a valid self-contained blob, recursively walk the graph to find unique external GGUF mmap-backed weight buffers and drop their resident pages while preserving the mapping address and tensor pointers. On Linux use page-aligned madvise(MADV_DONTNEED). On Windows, where there is no reliable per-range mapped-file equivalent, call VirtualUnlock for the identified range and trim the current process working set with SetProcessWorkingSetSize(-1, -1). Pages required later can fault back on demand. Keep the behavior disabled by default and document the physical-memory versus commit-memory tradeoff. Profiling reports the submitted GGUF range and model import duration. Validated with ReleaseOV llama-bench/llama-simple build and pp3072+tg512 warm mmap lazy import. With page release enabled, PrefillPeakWS dropped from about 5758 MiB to 3053 MiB while throughput remained comparable; PeakCommit stayed about 3331 MiB.
…eights bank When set, prefill and decode self-contained blobs use one shared NPUW_WEIGHTS_BANK name (ggml-sc-shared-<pfp>-<dfp>) instead of separate per-phase banks. Since both decoders are built from the same model_weights map, NPUW dedups them into a single in-process bank; on warm import the identical (uid,tensor) pairs are content-verified and reused (patched NPUW weights_bank). Cuts resident compiled-weight footprint ~2GB (pp3072 warm: In-Use 7954->5871 MiB, WS 7891->5765) with prefill throughput unchanged.
The shared weights bank (GGML_OPENVINO_NPU_SHARED_BANK) supersedes both: it deduplicates the prefill/decode compiled-weight copy directly, so lazy per-phase import and GGUF-page release no longer reduce steady memory. Remove the two env vars, the self-contained lazy-import branches (defaulting to the eager path that imports both blobs), the release_external_weight_pages helper, and the now-unused ggml_openvino_buffer_is_external / _release_external_pages buffer APIs.
In strict self-contained warm-mmap mode weights come via buffer_from_host_ptr, so every alloc_buffer is graph scratch/KV. The large prefill compute buffer is never touched on the host by the on-device fused OV model, yet the NPU L0 host tensor (create_host_tensor + memset) makes all ~2.4 GiB resident. For buffers >= 1 GiB in this mode, reserve demand-zero pages (no L0, no memset) so they stay non-resident; KV/small buffers keep the L0 host tensor for zero-copy import. pp3072 warm mmap + shared bank: prefill peak WS 5765 -> 3314 MiB (-43%), In-Use delta ~5940 -> ~3232 MiB; prefill 1256.7 -> 1252.9 t/s and decode 28.27 -> 27.96 t/s (both flat). Commit charge unchanged (demand-paged).
The static graph built the sliding-window KQ mask at the full context length, while the windowed layers bind their (smaller) window KV cache. With gemma-4 at -c 4096 -ub 1024 the window cache is 1536 cells, so SDPA shape inference failed ([1,1,1024,4096] mask vs [1,8,1536,256] K/V) and every prompt errored out. With GGML_OPENVINO_NPU_KV_SLICE the prefill mask was meant to use attention_size_swa, but the check matched "swa" in the ggml tensor name, and llama.cpp names both masks "attn_inp_kq_mask"; the windowed layers got the full-attention sliced length instead ([1,1,1024,256] vs [1,8,1024,256]). Identify the sliding-window mask by tensor identity (is_swa_mask) and size it to ctx_per_seq_swa, both in the graph input shape and in the runtime mask padding.
make_dirs() mkdir'ed every path prefix, including the drive root "D:", which fails with EACCES (not EEXIST) on Windows. Any absolute cache path therefore logged "could not create model cache dir ... caching disabled" and silently ran without GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. Skip the drive root.
…red code Mechanical refactor, no behavior change (step 1 of the NPU/MoE architecture plan): * openvino/pipelines/: TranslateSession::apply_transformations() now registers common passes, then the device's passes, then the execution-mode passes (stateful KV, static shapes), selected by a per-graph GraphTraits. The GPU MoE fusions and the GPU eltwise-rank workaround move to pipelines/gpu.cpp; CPU and NPU pipelines are empty placeholders for device lowerings. Pass order is unchanged. * npu/npu_config.*: the NPU (NPUW) compile config built at backend init, the NPUW strip for cacheless graphs and the per-graph static prefill/decode variants (HFA prefill, decode NPUW_UNFOLD_IREQS) move out of ggml-openvino-extra.cpp and utils.cpp. * npu/npu_quant_policy.*: the NPU requantization rules (requant policy, token_embd Q6_K, keep-Q4_0) move out of ggml_openvino_get_requant_type(). Verified identical output before/after on OV-CPU, GPU and NPU (gemma-4 E4B).
KV-cache slicing (bind the KV cache, and size the KQ masks, at the attended
length instead of the full context) existed twice: on by default for dynamic
graphs (GGML_OPENVINO_DISABLE_KV_SLICE to turn it off) and opt-in for static
prefill (GGML_OPENVINO_NPU_KV_SLICE). The static mode is not NPU-specific; it
applies to any static graph, including CPU/GPU with GGML_OPENVINO_FORCE_STATIC.
Describe it as one mechanism with two modes behind GGML_OPENVINO_KV_SLICE
(unset = each mode's default, 0 = off, 1 = on) and two predicates,
ggml_openvino_{dynamic,static}_kv_slice_enabled(), used by the KV binding,
the static graph shapes, the mask padding and the cache keys. The two old
variables remain as deprecated aliases.
…inputs dev_backend_openvino added shape_source/shape_axis to ModelExtraInputInfo; the HFA past-KV and mask initializers still listed four fields, which GCC rejects under -Werror=missing-field-initializers.
…ersion - Convert the static prefill and decode models one after the other, both before either is compiled. The decoders share the weight nodes; converting on two threads crashed intermittently inside libopenvino, and converting decode after the prefill compile gave wrong output on Bonsai-8B. - Compile prefill and decode in parallel only with GGML_OPENVINO_NPU_PARALLEL_COMPILE=1, since it raises the host memory peak. - Unfold NPUW function calls on the decode graph only with GGML_OPENVINO_NPU_DECODE_UNFOLD=1. The unfold default gave wrong output on gemma-4-E4B and slowed decode on several small models. - Stream NPU weight requantization only with GGML_OPENVINO_REDUCE_COMPILE_MEM or GGML_OPENVINO_MEMORY_OPTIMIZE, as on the other devices. - Fix a -Werror=missing-field-initializers build error in utils.cpp. - Declare ggml_openvino_get_npu_requant_type() only in npu/npu_quant_policy.h. - Docs: document the new switches, deprecate stateful execution, show stateless-only results in the validated models table, and update the validation system.
ravi9
force-pushed
the
dev_backend_openvino
branch
from
October 24, 2026 06:11
c5cd082 to
947be59
Compare
swathiananth
force-pushed
the
llamacpp_OV_pad_issue
branch
from
October 26, 2026 20:00
b02f359 to
fe4eff7
Compare
swathiananth
marked this pull request as ready for review
October 26, 2026 20:00
ravi9
force-pushed
the
dev_backend_openvino
branch
from
October 26, 2026 21:41
674af4a to
9e01c2f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Stateful execution drops leading dims, so PAD inputs can have rank < 4 while pads were always built with 4 entries, failing OpenVINO shape validation ("length of pads_begin mismatches with rank of input").
Trim the dropped leading pads for both regular and circular PAD.
Additional information
Summary
With
GGML_OPENVINO_STATEFUL_EXECUTION=1, Qwen3-VL models fail on the very first graph evaluation on the OpenVINO backend. OpenVINO shape inference rejects aPadnode because the pad vectors have 4 entries while the padded tensor has rank 3.The fix is in
ggml/src/ggml-openvino/openvino/op/pad.cpp. When the PAD input has rank < 4, the translator now drops the leading pad entries for the missing dimensions (after checking they are zero), so the pad vectors always match the inputrank. This applies to both regular (constant) PAD and circular PAD.
Symptom
Command (Windows, Intel GPU):
Output:
The model loads, but prompt evaluation fails and no tokens are generated (exit code 1).
Root cause
Where the PAD comes from
Qwen3-VL uses deepstack layers, so its input embedding width (
n_embd_inp) is larger than the model width (n_embd):// src/llama-hparams.cpp n_embd_inp = n_embd + n_embd * n_deepstack_layers;When the graph is built from text tokens, the token embeddings (width
n_embd=2560) are zero-padded along
ne0up ton_embd_inp:So the PAD input is the
GET_ROWSresult on the token embedding table, which the OpenVINO frontend translates into aGather.That is the
Gather_2596 ... f32[1,?,2560]in the error message.Why the ranks disagree
GGML tensors are always 4-D. A GGML PAD carries 8 pad values in
op_params(begin/end forne0..ne3), and the translator mapped them to OpenVINO axis order (OpenVINO axis 0 = GGMLne3):pads_begin = {pads[6], pads[4], pads[2], pads[0]}; // always 4 entries pads_end = {pads[7], pads[5], pads[3], pads[1]};This silently assumed the OpenVINO input also had rank 4. In stateful execution
the frontend drops leading singleton dimensions, so the embedding
Gatherproducesa rank-3 tensor
[1, n_tokens, 2560].ov::op::v1::Padrequireslen(pads_begin) == len(pads_end) == rank(input), so validation fails with"expect 3, but got 4".
Circular PAD (
translate_circular_pad) had the same assumption: it indexedpads_begin[axis]foraxisin[0, rank). With a rank-3 input that would applyGGML
ne3's pads to OpenVINO axis 0,ne2's to axis 1, and so on, which is thewrong axis rather than an error.
The fix
Dropped dimensions are always the leading ones (GGML
ne3, thenne2, ...), so the pads for the axes the input still has are the trailing entries of the 4-element vectors. The fix aligns the pads to the trailing input axes.Regular (constant) PAD:
translate_padinput.get_partial_shape()),not from the GGML shape, because that is what
Padvalidates against.constso they can be trimmed.For the failing node: input rank 3,
dropped = 1; pads{0,0,0,P}/{0,0,0,0}(P =n_embd_inp - n_embd) become{0,0,P}/{0,0,0},which pads the last axis (GGML
ne0) as intended.Circular PAD:
translate_circular_padSame idea: skip
droppedleading entries so each OpenVINO axis gets the pads ofits matching GGML dimension.
Why the zero check is safe
A dimension can only be dropped if it is a singleton, and a PAD that grows that dimension would make the output rank-4 again, which the frontend would not have squeezed. So in valid graphs the dropped pads are always zero. If that ever stops being true, the check fails with a clear conversion error instead of silently padding the wrong axis.
##Testing
Qwen3-VL-4B-Instruct Q4_K_M: Testing Summary
These results are from Intel Arc Pro B70 (32 GB) on Windows 11 with OpenVINO 2026.4, running on the GPU. The base build is commit 52e2420.
1. The Pad bug (stateful execution)
llama-completion -f prompt_1024.txt -c 1382 -n 128 -st -ngl -1 -no-cnv --ignore-eos, withGGML_OPENVINO_STATEFUL_EXECUTION=1:Response.txt)Perplexity of the generated response (
-c 64 -b 64 -np 1, 2 chunks):CL_OUT_OF_RESOURCES)2. Stateless execution
Completion (greedy,
-fa on): exit 0. The output reads "The sky appears blue because of a phenomenon called Rayleigh scattering, which occurs when sunlight interacts wi…" (truncated in the log).Decode (tg128, tokens/s, median ± half the spread):
Prefill (tokens/s):
Can not allocate 8388608 bytes for USM Device)Accuracy (GPU against the CPU backend; ctx 1024; perplexity on wikitext-2 over 16 chunks):
3. The pp8192 failure: base vs. fix (same session, back-to-back)
At 8K the model uses about 32 GB on both builds, the card's full capacity, so the test passes or fails by chance. When it does pass, throughput drops sharply, from about 6,000 t/s at pp4096 to about 2,500 at pp8192, with ±50% spread. That points to memory spilling past the GPU's dedicated memory.
Conclusions
Requirements