Repository navigation
Update the llama.cpp to utilize OV frontend op translations - #342
Open
preetha-intel wants to merge 28 commits into
Open
preetha-intel wants to merge 28 commits into
preetha-intel wants to merge 28 commits into
Conversation
When the SWA cache size matches the full cache size, identify SWA layers by the number of layers using each mask instead of cache size; the larger group is SWA, and a tie leaves the layers unclassified.
Give SWA KV cache index tensors unique input names (a _swa suffix, like the masks) to prevent them from being merged with the full-attention cache indices.
Two attention masks do not imply SWA: DeepSeek sparse attention uses two full-attention masks with the same name. When cache extents match, classify the larger mask group as SWA only once its mask has fewer keys per row than the other mask. Keep the classification per KV cache buffer so new sequences reuse the same graph, and include swa_layers when checking graph reuse.
Waiting for the window to appear in the mask caused a mid-sequence graph recompile (~9 s for Gemma-4 at -c 1024 after position 512). SWA classification already gives each mask group its own mask, KV write indices, and attention size, which is correct whether or not the mask is a sliding window. Stateful mask rebuild recovers the window from the mask, or the causal mask for full attention. Classify the larger of two mask groups from the first graph and drop the per-KV-buffer bookkeeping.
Upstream ggml-org#29622 adds a mixed token/embd branch to every input embedding graph through ggml_build_forward_select(). Its nodes are not flagged for compute, but the backend translated them anyway, and the DUP in that branch was unsupported, so the scheduler split the graph and passed the embeddings across the split with a fixed token count. The first single-token decode then failed (test-thread-safety on CPU and GPU). Build the OV model from the compute nodes only, and translate a same-type contiguous DUP like CONT so the graph stays on one backend.
ggml-org#29622 also moves the per-token embedding scale (gemma3, gemma3n, gemma4) into a new [1, n_tokens] input. Give it a dynamic token dim and pad it per chunk on the static (NPU) path.
Op tests build Q4_1/Q4_K weights as u4 with an f16 zero point. The GPU plugin fails to compile that form for some row counts with "clFinish, error code: -5 CL_OUT_OF_RESOURCES", which aborts test-backend-ops on the MUL_MAT cases added in ggml-org#29869 (e.g. m=1000, n=2, k=1024). Model weights use a u4 zero point and are not affected. Report these cases as unsupported on GPU until the plugin is fixed. Op tests check support before allocating, so the check matches unbound weights only; model loading probes with a dummy buffer and keeps its weights on the GPU.
translate_fill always built an f32 constant, so an f16 FILL produced f32 data and the copy back overran the f16 output buffer. Use the output type for the constant.
Quantized inputs are dequantized when translated, so the backend cannot write a quantized CONCAT output. Report it as unsupported, as for CPY to a quantized type.
ggml-org#29856 changed build_rs to gather all recurrent states with one GET_ROWS on the s_copy leaf and take the ubatch and extra states as views of it. The stateful path matched only the previous form, a GET_ROWS per view of s_copy, so Qwen3.5 failed with stateful execution on CPU and GPU ("is_axis_valid(axis, r)" in a Concat). For a single-slot cache, treat the GET_ROWS on the s_copy leaf as the active-state gather, keep the rank-4 layout of reshapes that read a view of it, and map the copy of the empty extra-state view to the single-slot remainder writeback. Do not warn about the dynamic dim of empty views.
…randRanks The pass unsqueezes the lower-rank operand of an Add/Multiply/Subtract whose operand ranks differ. In gemma-3 the lower-rank operand of the post-attention residual add is the norm output, and unsqueezing it makes the GPU plugin compute the layer wrongly: gemma-3 returns empty answers on GPU with stateful execution. Skip the rewrite when the lower-rank operand is an RMS norm output.
Dynamic CPU/GPU graphs previously passed n_seq_active and attention_size(_swa) as separate inputs. Derive these values directly from the KQ mask shape using ShapeOf + Gather inside the model. This allows the GPU plugin to infer the Q reshape and KV slice shapes from shape inference rather than runtime input values. Q's head size also becomes static, enabling the fused SDPA kernel. GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK=1 restores the previous separate-input behavior. The fused SDPA path also requires two supporting changes: - Clamp the GPU mask to -30000. A key block that is entirely -inf for a row produces NaNs in the fused kernel. - Restate the head count and head size after the KV slice. The plugin loses all slice dimensions when the end index is dynamic, causing SDPA to fall back to the reference kernel. Additional graph and quantization improvements: - Skip the identity reshape of rows in SET_ROWS. - Share a single Split for one-element view slices in dynamic GPU graphs (e.g. Gemma-4 per-layer embeddings), replacing one StridedSlice per layer. - Add GGML_OPENVINO_REQUANT_EMBD=native (off by default) to preserve the file's native quantization for token_embd/output instead of requantizing to Q8_0_C. - Include both environment settings in the graph and model cache keys and document them.
The dynamic GPU path reused a Split output for any single-element view by deriving the chunk index from offset / stride. This produced incorrect results for views that were not chunk-aligned or whose remaining dimensions did not preserve the source strides, as seen in test-backend-ops CONT with use_view_slice. Restrict Split reuse to cases where the view exactly matches a split chunk, meaning the offset is aligned to the split stride and the remaining non-unit dimensions preserve the original source strides. Gemma-4 per-layer embedding slices continue to use the shared Split path.
Add a TODO above the equal-extent fallback to document that the larger mask group is currently assumed to be the SWA group. Since stateless graphs only require the two groups to remain distinct, this heuristic may only be necessary for stateful execution and could potentially be restricted to that path.
ggml-openvino: Fix SWA layer detection and KV write indices (Gemma-4-E2B past 512 tokens)
Initialize the newly added `shape_source` and `shape_axis` fields in the `ModelExtraInputInfo` brace initializer used by `add_extra_inputs()`. Without initializers for these fields, GCC emits `-Wmissing-field-initializers`, causing CI to fail with `LLAMA_FATAL_WARNINGS`.
The option identifies embedding and output weights by their tensor names, which are defined by core llama.cpp. Remove it from this PR; preserving the file quantization of these weights will be addressed in a separate change that identifies them by their roles in the graph.
ggml-openvino: derive attention sizes from the KQ mask shape
Collaborator
|
Thanks for contributing! Development happens on |
The manual GQA attention path reshaped Q, K, and V to 5-D so that query heads sharing a KV head could broadcast against it. The GPU plugin cannot fuse this graph: it materializes the attention scores and copies the transposed K and V caches for every layer. Instead, stack the query heads for each KV head along the sequence axis (Q [B, H, S, D] -> [B, H_kv, factor * S, D]) and tile the mask accordingly. This keeps the graph 4-D from the MatMuls through the Softmax, allowing the GPU plugin to fuse the attention into its SDPA kernel and fold the K and V transposes into the fused operation. Clamp the mask to -30000, matching the SDPA path, because the fused kernel can return NaN when an entire block is masked with -inf. Models where the K or V head size differs from the Q head size now use the SDPA path, since the manual path assumes one head size.
ggml-openvino: keep manual GQA attention 4-D for SDPA fusion
In stateless GPU mode, bind the entire KV cache as a `[1, ctx, n_heads_kv, head_size]` tensor, with one token per row, instead of binding the first `n_kv` rows as a `[1, 1, ctx, n_heads_kv * head_size]` tensor. Write cache updates with `ScatterUpdate` along the token axis, and read the cache through `VariadicSplit` and `Transpose`. This allows the GPU plugin's `StatelessKVFusion` to replace cache reads and writes with in-place `StatelessKV` nodes, avoiding full-cache copies on every decode step and repeated memory pool allocations as the bound cache size grows. Set `GGML_OPENVINO_DISABLE_KV_4D=1` to retain the previous layout.
ggml-openvino: enable in-place 4-D stateless KV cache updates on GPU
Add GGML decoder to support OV GGUF Frontend.
Collaborator
|
Could we stage this migration behind an opt-in CMake flag, keeping the existing translations as the default for now? This change replaces a substantial translation and fusion layer. Retaining the legacy path temporarily would let us compare correctness, model/device coverage, and performance, while providing an easy rollback. The GGUF frontend dependency should only be required when the new path is enabled. Once validation gives us confidence, we can switch the default and remove the legacy implementation in a follow-up PR. @ravi9, what do you think about taking this incremental approach? |
Owner
|
Sure, an incremental approach with the opt-in CMake flag sounds good. |
ravi9
force-pushed
the
dev_backend_openvino
branch
from
October 20, 2026 18:50
61a6258 to
df21ad8
Compare
Granite's residual and attention multipliers leave the f16 range of the GPU and NPU plugins before the scale is applied, so Granite 4.0 350M/1B printed '?'. The frontend provides the rewrite (PushScaleBeforeMatMul and the "prescale_q" FLASH_ATTN_EXT attribute) but does not apply it itself, as it is only useful when the device executes in f16. - Register PushScaleBeforeMatMul as a frontend extension and set prescale_q when the device is GPU or NPU. CPU runs in f32 and is left unchanged. - GGML_OPENVINO_DISABLE_SCALE_REORDER=1 turns it off. It is part of the compiled-model cache key and listed in the env settings and docs.
github-actions
Bot
added
the
documentation
ravi9
force-pushed
the
dev_backend_openvino
branch
4 times, most recently
from
October 29, 2026 17:37
be796d8 to
25e3135
Compare
|
hey @ravi9 and @cavusmustafa!! Thank you!! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add GGML decoder to support OV GGUF Frontend.
Overview
This PR removes the legacy in-tree OpenVINO translation layer from ggml-openvino and moves the backend onto the OpenVINO GGUF frontend (openvino::frontend::gguf).
Until now the backend carried its own copy of the GGML→OpenVINO conversion: a decoder interface, node context, per-op translators, graph-fusion passes and a translate session, all under ggml/src/ggml-openvino/openvino/.
After this change, the frontend owns the generic op translations and passes. The backend keeps a GgmlOvDecoder bridge that exposes the ggml graph to the frontend, plus the runtime pieces (device handling, KV cache, compile and cache logic).
Changes
CMake changes
Openvino Frontend changes PR : openvinotoolkit/openvino#38769