ggml-openvino: one KV-slice switch for dynamic and static graphs

KV-cache slicing (bind the KV cache, and size the KQ masks, at the attended
length instead of the full context) existed twice: on by default for dynamic
graphs (GGML_OPENVINO_DISABLE_KV_SLICE to turn it off) and opt-in for static
prefill (GGML_OPENVINO_NPU_KV_SLICE). The static mode is not NPU-specific; it
applies to any static graph, including CPU/GPU with GGML_OPENVINO_FORCE_STATIC.

Describe it as one mechanism with two modes behind GGML_OPENVINO_KV_SLICE
(unset = each mode's default, 0 = off, 1 = on) and two predicates,
ggml_openvino_{dynamic,static}_kv_slice_enabled(), used by the KV binding,
the static graph shapes, the mask padding and the cache keys. The two old
variables remain as deprecated aliases.

Author: anikulk

docs/backend/OPENVINO.md

@@ -728,10 +728,11 @@ Boolean flags follow a uniform convention: set to a **positive integer** (e.g. `
 | `GGML_OPENVINO_NPU_REQUANT_POLICY`| String    | `group-128` | NPU-only weight requantization layout. Use `group-128` for `Q4_0_128` or experimental `channel-wise` for `Q4_0_C`. Any other value fails during model loading. |
 | `GGML_OPENVINO_NPU_FAST_MASK`     | Boolean   | `0`        | NPU-only. Build the static prefill attention mask directly into the input tensor, skipping an intermediate host allocation + full-tensor copy per chunk. Numerically identical to the default path. |
 | `GGML_OPENVINO_NPU_L0_HOST_TENSORS`| Boolean  | `0`        | NPU-only. Back the KV-cache / compute buffers with Level Zero host memory so the plugin imports them without an extra host-to-device copy each infer, cutting prefill KV I/O time. Falls back to plain host memory if the L0 allocation is unavailable. |
-| `GGML_OPENVINO_NPU_KV_SLICE`      | Boolean   | `0`        | NPU-only. Compile static prefill graphs and bind prefill KV-cache tensors at the active attention length instead of the full context capacity. Decode retains its fixed full-context graph. The prefill model is recompiled when the aligned attention length changes. |
+| `GGML_OPENVINO_KV_SLICE`          | Boolean   | per mode   | Bind the KV cache, and size the KQ masks, at the attended length instead of the full context. Dynamic graphs (CPU/GPU): a view over the cache, no recompilation; on by default. Static graphs (NPU, or `GGML_OPENVINO_FORCE_STATIC`): the prefill graph is compiled at the aligned attended length and recompiled when it changes, decode keeps its full-context graph; off by default. Unset = each mode's default, `0` = off in both, `1` = on in both. |
+| `GGML_OPENVINO_NPU_KV_SLICE`      | Boolean   | `0`        | Deprecated alias: `1` turns static KV slicing on (`GGML_OPENVINO_KV_SLICE` wins when set). Not NPU-specific; applies to any static graph. |
 | `GGML_OPENVINO_STATEFUL_EXECUTION`| Boolean   | `0`        | Keep KV and supported recurrent caches inside the model. Single-slot CPU/GPU execution only.               |
 | `GGML_OPENVINO_DISABLE_CACHE`     | Boolean   | `0`        | Disable the in-process compiled-model / decoder cache (cache is on by default). Set to `1` to disable.      |
-| `GGML_OPENVINO_DISABLE_KV_SLICE`  | Boolean   | `0`        | Disable the KV-cache input-tensor slicing optimization (slicing is on by default on CPU/GPU). Set to `1` to disable. |
+| `GGML_OPENVINO_DISABLE_KV_SLICE`  | Boolean   | `0`        | Deprecated alias: `1` turns dynamic KV slicing off (`GGML_OPENVINO_KV_SLICE` wins when set). |
 | `GGML_OPENVINO_ENABLE_KV_4D`      | Boolean   | `0`        | GPU, stateless: bind the KV cache as `[1, ctx, n_heads_kv, head_size]` instead of `[1, 1, ctx, n_heads_kv * head_size]`, so the GPU plugin's `StatelessKVFusion` can update and read the cache in place. **Off by default**: it needs an OpenVINO build with openvino#37569 (2026.5 or a recent nightly) and is much slower on older releases (e.g. 2026.4.1). Set to a positive integer to enable. |
 | `GGML_OPENVINO_KV_SCATTER_ELEMENTS` | Boolean | `0`        | Use `ScatterElementsUpdate` instead of `ScatterUpdate` for non-stateful single-row KV writes. Intended for NPU decode paths where `ScatterUpdate` copies the full destination tensor to update one row. |
 | `GGML_OPENVINO_DISABLE_KV_STATE_RELAYOUT` | Boolean | `0`     | Disable the stateful KV-state sequence-axis relayout (relayout is on by default). It moves the KV state sequence axis from dim 1 to dim 2, so the GPU plugin can append new tokens in place instead of copying the whole state every token, and the reader side no longer transposes the whole accumulated state. Set to `1` to disable. |

ggml/src/ggml-openvino/ggml-decoder.cpp

@@ -1025,7 +1025,7 @@ std::pair<ModelParams, ComputeParams> GgmlOvDecoder::compute_llm_params(ggml_cgr
                 compute_params.mask_name = mask_name;
             }
             if (is_static) {
-                const bool slice_prefill = ggml_openvino_npu_kv_slice_enabled() && compute_params.input_len > 1;
+                const bool slice_prefill = ggml_openvino_static_kv_slice_enabled() && compute_params.input_len > 1;
                 if (!slice_prefill) {
                     compute_params.attention_size = model_params.ctx_per_seq;
                 }
@@ -1048,7 +1048,7 @@ std::pair<ModelParams, ComputeParams> GgmlOvDecoder::compute_llm_params(ggml_cgr
             node->src[0]->src[0]->op == GGML_OP_VIEW) {
             compute_params.attention_size = node->ne[0];
             compute_params.mask_name.clear();
-            if (is_static && !(ggml_openvino_npu_kv_slice_enabled() && compute_params.input_len > 1)) {
+            if (is_static && !(ggml_openvino_static_kv_slice_enabled() && compute_params.input_len > 1)) {
                 compute_params.attention_size = model_params.ctx_per_seq;
             }
         }
@@ -1204,7 +1204,7 @@ ov::PartialShape GgmlOvDecoder::get_graph_input_shape(const ggml_tensor * op,
             // Both masks are named "attn_inp_kq_mask", so tell them apart by tensor identity.
             const bool swa_mask = is_swa_mask(input);
             int attention_size = swa_mask ? m_model_params.ctx_per_seq_swa * m_model_params.n_seq : m_model_params.ctx;
-            if (m_is_prefill && ggml_openvino_npu_kv_slice_enabled()) {
+            if (m_is_prefill && ggml_openvino_static_kv_slice_enabled()) {
                 attention_size = swa_mask ? m_compute_params.attention_size_swa : m_compute_params.attention_size;
             }
             input_shape = ov::PartialShape{1, 1, m_is_prefill ? m_prefill_chunk_size : 1, attention_size};
@@ -1230,7 +1230,7 @@ ov::PartialShape GgmlOvDecoder::get_graph_input_shape(const ggml_tensor * op,
         if (!m_is_static && !is_flat_kv) {
             // do not fix ctx size to make llama-bench work across test params
             input_shape[2] = -1;
-        } else if (m_is_prefill && ggml_openvino_npu_kv_slice_enabled()) {
+        } else if (m_is_prefill && ggml_openvino_static_kv_slice_enabled()) {
             const auto layer = extract_layer_from_name(name);
             if (layer.has_value() && !is_swa_layer(*layer) && m_compute_params.attention_size > 0 &&
                 m_compute_params.attention_size < input_shape[2].get_length()) {

ggml/src/ggml-openvino/ggml-openvino-extra.cpp

@@ -160,6 +160,7 @@ void ggml_openvino_device_config::init() {
         "GGML_OPENVINO_GDN_Q_SCALE",
         "GGML_OPENVINO_NPU_L0_HOST_TENSORS",
         "GGML_OPENVINO_NPU_KV_SLICE",
+        "GGML_OPENVINO_KV_SLICE",
         "GGML_OPENVINO_NPU_HFA_PREFILL",
         "GGML_OPENVINO_SELF_CONTAINED_BLOB",
         "GGML_OPENVINO_SELF_CONTAINED_MMAP",
@@ -341,7 +342,28 @@ bool ggml_openvino_is_gpu() {
     return has_prefix(ggml_openvino_get_device_name(), "GPU");
 }
 
-bool ggml_openvino_npu_kv_slice_enabled() {
+// -1: GGML_OPENVINO_KV_SLICE unset, use the mode's default (or its deprecated alias)
+static int kv_slice_setting() {
+    const char * value = ggml_openvino_getenv_str("GGML_OPENVINO_KV_SLICE");
+    if (value == nullptr || *value == '\0') {
+        return -1;
+    }
+    return ggml_openvino_getenv_int("GGML_OPENVINO_KV_SLICE") != 0 ? 1 : 0;
+}
+
+bool ggml_openvino_dynamic_kv_slice_enabled() {
+    const int setting = kv_slice_setting();
+    if (setting >= 0) {
+        return setting != 0;
+    }
+    return ggml_openvino_getenv_int("GGML_OPENVINO_DISABLE_KV_SLICE") == 0;
+}
+
+bool ggml_openvino_static_kv_slice_enabled() {
+    const int setting = kv_slice_setting();
+    if (setting >= 0) {
+        return setting != 0;
+    }
     return ggml_openvino_getenv_int("GGML_OPENVINO_NPU_KV_SLICE") != 0;
 }
 

ggml/src/ggml-openvino/ggml-openvino-extra.h

@@ -119,7 +119,15 @@ bool ggml_openvino_is_npu();
 
 // NPU-only opt-in performance toggles (all default off). Centralized here so each
 // env-var name lives in one place; see docs/backend/OPENVINO.md for semantics.
-bool ggml_openvino_npu_kv_slice_enabled();
+// KV-cache slicing: bind the KV cache, and size the KQ masks, at the attended length instead of the full context.
+// One mechanism, two modes:
+//   dynamic graphs (CPU/GPU): the KV input is a view over the cache; the graph is not recompiled. On by default.
+//   static graphs (NPU, or any device with GGML_OPENVINO_FORCE_STATIC): the prefill graph is compiled at the aligned
+//   attended length and recompiled when it changes; decode keeps the full context. Off by default.
+// GGML_OPENVINO_KV_SLICE: unset = the default of each mode, 0 = off in both, 1 = on in both.
+// Deprecated aliases: GGML_OPENVINO_DISABLE_KV_SLICE=1 (dynamic off), GGML_OPENVINO_NPU_KV_SLICE=1 (static on).
+bool ggml_openvino_dynamic_kv_slice_enabled();
+bool ggml_openvino_static_kv_slice_enabled();
 bool ggml_openvino_npu_fast_mask_enabled();
 bool ggml_openvino_npu_l0_host_tensors_enabled();
 bool ggml_openvino_npu_hfa_prefill_enabled();

ggml/src/ggml-openvino/model-cache.cpp

@@ -174,6 +174,7 @@ static const char * cache_settings[] = {
     "GGML_OPENVINO_REQUANT_KQUANT",
     "GGML_OPENVINO_NATIVE_SOFTPLUS",
     "GGML_OPENVINO_DISABLE_KV_SLICE",
+    "GGML_OPENVINO_KV_SLICE",
     "GGML_OPENVINO_MANUAL_GQA_ATTN",
     "GGML_OPENVINO_STATEFUL_EXECUTION",
     "GGML_OPENVINO_DISABLE_KV_STATE_RELAYOUT",

ggml/src/ggml-openvino/utils.cpp

@@ -56,13 +56,12 @@ namespace {
 std::optional<ov::Tensor> try_make_kv_sliced_tensor(const std::shared_ptr<GgmlOvDecoder> & ggml_decoder,
                                                     const std::string & name,
                                                     const ggml_tensor * ggml_tensor) {
-    static const bool kv_slice_disabled = ggml_openvino_getenv_int("GGML_OPENVINO_DISABLE_KV_SLICE");
-    if (kv_slice_disabled) {
-        return std::nullopt;
-    }
-    const bool static_npu_slice =
-        ggml_decoder->is_static() && ggml_decoder->m_is_prefill && ggml_openvino_npu_kv_slice_enabled();
-    if ((ggml_decoder->is_static() && !static_npu_slice) || ggml_decoder->is_stateful()) {
+    // Static graphs bake the sliced length into the prefill graph (see ggml_openvino_static_kv_slice_enabled);
+    // decode keeps its full-context graph.
+    const bool slice = ggml_decoder->is_static() ?
+                           ggml_decoder->m_is_prefill && ggml_openvino_static_kv_slice_enabled() :
+                           ggml_openvino_dynamic_kv_slice_enabled();
+    if (!slice || ggml_decoder->is_stateful()) {
         return std::nullopt;
     }
     if (ggml_tensor->op != GGML_OP_NONE || ggml_tensor->view_src != nullptr) {
@@ -142,7 +141,7 @@ std::optional<ov::Tensor> try_make_kv_4d_tensor(const std::shared_ptr<GgmlOvDeco
 // Column count of a static KQ mask input; must match GgmlOvDecoder::get_graph_input_shape().
 static size_t get_static_attention_size(std::shared_ptr<GgmlOvDecoder> ggml_decoder, const std::string & name) {
     const bool is_swa = ggml_decoder->is_swa_mask(ggml_decoder->get_input_ggml_tensor(name));
-    if (ggml_decoder->m_is_prefill && ggml_openvino_npu_kv_slice_enabled()) {
+    if (ggml_decoder->m_is_prefill && ggml_openvino_static_kv_slice_enabled()) {
         const auto params = ggml_decoder->get_compute_params();
         const int attention_size = is_swa ? params.attention_size_swa : params.attention_size;
         if (attention_size > 0) {
@@ -165,7 +164,7 @@ static uint64_t ggml_openvino_model_cache_extra_cfg(bool stateful, bool recurren
     uint64_t extra_cfg = 1;  // Graph-ordinal port names (invalidate older disk-cache blobs).
     extra_cfg = extra_cfg * 131 + (stateful ? (recurrent ? 2u : 1u) : 0u);
     extra_cfg = extra_cfg * 131 + (ggml_openvino_reduce_compile_mem_enabled() ? 1u : 0u);
-    extra_cfg = extra_cfg * 131 + (ggml_openvino_getenv_int("GGML_OPENVINO_DISABLE_KV_SLICE") ? 1u : 0u);
+    extra_cfg = extra_cfg * 131 + (ggml_openvino_dynamic_kv_slice_enabled() ? 0u : 1u);
     extra_cfg = extra_cfg * 131 + (ggml_openvino_getenv_int("GGML_OPENVINO_ENABLE_KV_4D") > 0 ? 1u : 0u);
     extra_cfg = extra_cfg * 131 + (manual_gqa_enabled ? 1u : 0u);
     extra_cfg = extra_cfg * 131 + (ggml_openvino_getenv_int("GGML_OPENVINO_PROFILING") >= 2 ? 1u : 0u);
@@ -1736,7 +1735,7 @@ enum ggml_status ov_graph_compute_static(ggml_cgraph * cgraph, const std::shared
         ggml_decoder = entry->ptr;
         old_m_params = ggml_decoder->get_model_params();
         cache_hit = old_m_params.can_reuse_statically(m_params);
-        if (cache_hit && is_prefill && ggml_openvino_npu_kv_slice_enabled()) {
+        if (cache_hit && is_prefill && ggml_openvino_static_kv_slice_enabled()) {
             const auto old_c_params = ggml_decoder->get_compute_params();
             cache_hit = old_c_params.attention_size == c_params.attention_size &&
                         old_c_params.attention_size_swa == c_params.attention_size_swa;
@@ -1814,7 +1813,7 @@ enum ggml_status ov_graph_compute_static(ggml_cgraph * cgraph, const std::shared
             GGML_LOG_DEBUG("ggml-openvino: shared compiled model HIT (static)\n");
         } else {
             auto decode_c_params = c_params;
-            if (ggml_openvino_npu_kv_slice_enabled()) {
+            if (ggml_openvino_static_kv_slice_enabled()) {
                 decode_c_params.attention_size = m_params.ctx_per_seq;
                 decode_c_params.attention_size_swa = m_params.ctx_per_seq_swa;
             }
@@ -1841,7 +1840,7 @@ enum ggml_status ov_graph_compute_static(ggml_cgraph * cgraph, const std::shared
                 base_extra = base_extra * 131 + static_cast<uint64_t>(ggml_openvino_getenv_int("GGML_OPENVINO_TOKEN_EMBD_I8"));
                 base_extra = base_extra * 131 + static_cast<uint64_t>(ggml_openvino_getenv_int("GGML_OPENVINO_TOKEN_EMBD_I4"));
                 base_extra = base_extra * 131 + static_cast<uint64_t>(ggml_openvino_getenv_int("GGML_OPENVINO_NPU_KEEP_Q4_0"));
-                base_extra = base_extra * 131 + static_cast<uint64_t>(ggml_openvino_getenv_int("GGML_OPENVINO_NPU_KV_SLICE"));
+                base_extra = base_extra * 131 + (ggml_openvino_static_kv_slice_enabled() ? 1u : 0u);
                 base_extra = ggml_openvino_model_cache_mix_string(
                     base_extra, ggml_openvino_getenv_str("GGML_OPENVINO_NPU_CONFIG"));
                 base_extra = ggml_openvino_model_cache_mix_string(