ggml-openvino: unify KV slicing under GGML_OPENVINO_KV_SLICE, on by default

Replace ggml_openvino_dynamic_kv_slice_enabled() and ggml_openvino_static_kv_slice_enabled() with a single ggml_openvino_kv_slice_enabled(). KV slicing binds the KV cache, and sizes the KQ masks, at the attended length instead of the full context.

GGML_OPENVINO_KV_SLICE now defaults to 1 on CPU, GPU and NPU; set it to 0 to disable. Dynamic graphs (CPU/GPU) bind a view over the cache without recompiling. Static graphs (NPU or GGML_OPENVINO_FORCE_STATIC) compile prefill at the aligned attended length and recompile when it changes; decode keeps its full-context graph.

Remove the deprecated aliases GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_NPU_KV_SLICE from the env var list, the model-cache settings, docs/backend/OPENVINO.md and the backend README.

Behavior change: static KV slicing is now on by default on NPU, so prefill compiles one graph per distinct n_kv (multiples of 256).

Author: zhaixuejun1993

docs/backend/OPENVINO.md

@@ -726,11 +726,9 @@ Boolean flags follow a uniform convention: set to a **positive integer** (e.g. `
 | `GGML_OPENVINO_NPU_REQUANT_POLICY`| String    | `group-128` | NPU-only weight requantization layout. Use `group-128` for `Q4_0_128` or experimental `channel-wise` for `Q4_0_C`. Any other value fails during model loading. |
 | `GGML_OPENVINO_NPU_FAST_MASK`     | Boolean   | `0`        | NPU-only. Build the static prefill attention mask directly into the input tensor, skipping an intermediate host allocation + full-tensor copy per chunk. Numerically identical to the default path. |
 | `GGML_OPENVINO_NPU_L0_HOST_TENSORS`| Boolean  | `0`        | NPU-only. Back the KV-cache / compute buffers with Level Zero host memory so the plugin imports them without an extra host-to-device copy each infer, cutting prefill KV I/O time. Falls back to plain host memory if the L0 allocation is unavailable. |
-| `GGML_OPENVINO_KV_SLICE`          | Boolean   | per mode   | Bind the KV cache, and size the KQ masks, at the attended length instead of the full context. Dynamic graphs (CPU/GPU): a view over the cache, no recompilation; on by default. Static graphs (NPU, or `GGML_OPENVINO_FORCE_STATIC`): the prefill graph is compiled at the aligned attended length and recompiled when it changes, decode keeps its full-context graph; off by default. Unset = each mode's default, `0` = off in both, `1` = on in both. |
-| `GGML_OPENVINO_NPU_KV_SLICE`      | Boolean   | `0`        | Deprecated alias: `1` turns static KV slicing on (`GGML_OPENVINO_KV_SLICE` wins when set). Not NPU-specific; applies to any static graph. |
+| `GGML_OPENVINO_KV_SLICE`          | Boolean   | `1`        | Bind the KV cache, and size the KQ masks, at the attended length instead of the full context. Dynamic graphs (CPU/GPU): a view over the cache, no recompilation. Static graphs (NPU, or `GGML_OPENVINO_FORCE_STATIC`): the prefill graph is compiled at the aligned attended length and recompiled when it changes, decode keeps its full-context graph. Set to `0` to disable. |
 | `GGML_OPENVINO_STATEFUL_EXECUTION`| Boolean   | `0`        | **Deprecated**, to be removed; use stateless execution (`0`). Keep KV and supported recurrent caches inside the model. Single-slot CPU/GPU execution only. |
 | `GGML_OPENVINO_DISABLE_CACHE`     | Boolean   | `0`        | Disable the in-process compiled-model / decoder cache (cache is on by default). Set to `1` to disable.      |
-| `GGML_OPENVINO_DISABLE_KV_SLICE`  | Boolean   | `0`        | Deprecated alias: `1` turns dynamic KV slicing off (`GGML_OPENVINO_KV_SLICE` wins when set). |
 | `GGML_OPENVINO_ENABLE_KV_4D`      | Boolean   | `0`        | GPU, stateless: bind the KV cache as `[1, ctx, n_heads_kv, head_size]` instead of `[1, 1, ctx, n_heads_kv * head_size]`, so the GPU plugin's `StatelessKVFusion` can update and read the cache in place. **Off by default**: it needs an OpenVINO build with openvino#37569 (2026.5 or a recent nightly) and is much slower on older releases (e.g. 2026.4.1). Set to a positive integer to enable. |
 | `GGML_OPENVINO_KV_SCATTER_ELEMENTS` | Boolean | `0`        | Use `ScatterElementsUpdate` instead of `ScatterUpdate` for non-stateful single-row KV writes. Intended for NPU decode paths where `ScatterUpdate` copies the full destination tensor to update one row. |
 | `GGML_OPENVINO_DISABLE_KV_STATE_RELAYOUT` | Boolean | `0`     | Disable the stateful KV-state sequence-axis relayout (relayout is on by default). It moves the KV state sequence axis from dim 1 to dim 2, so the GPU plugin can append new tokens in place instead of copying the whole state every token, and the reader side no longer transposes the whole accumulated state. Set to `1` to disable. |

ggml/src/ggml-openvino/README.md

@@ -7,7 +7,7 @@ One directory can hold blobs for different models and compilation settings. Run
 ```sh
 GGML_OPENVINO_DEVICE=GPU \
 GGML_OPENVINO_NATIVE_SOFTPLUS=1 \
-GGML_OPENVINO_DISABLE_KV_SLICE=1 \
+GGML_OPENVINO_KV_SLICE=0 \
 GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all \
 GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR=/path/to/qwen-cache \
 ./build/ReleaseOV/bin/llama-bench -m /path/to/model.gguf -r 1
@@ -20,7 +20,7 @@ On later runs, supply the same compilation settings and enable `GGML_OPENVINO_CO
 ```sh
 GGML_OPENVINO_DEVICE=GPU \
 GGML_OPENVINO_NATIVE_SOFTPLUS=1 \
-GGML_OPENVINO_DISABLE_KV_SLICE=1 \
+GGML_OPENVINO_KV_SLICE=0 \
 GGML_OPENVINO_REQUANT_KQUANT=q4_asym64_all \
 GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR=/path/to/qwen-cache \
 GGML_OPENVINO_COMPILED_MODEL_CACHE_ONLY=1 \
@@ -29,7 +29,7 @@ GGML_OPENVINO_COMPILED_MODEL_CACHE_ONLY=1 \
 
 Cache-only mode allocates backend address space without filling weight pages. On Windows, this also uses system commit capacity. The model-buffer size in the loader log is this virtual size. Weight uploads only record source identity; they do not read or requantize the weights. Graph conversion and compilation are skipped. Runtime buffers are still allocated and populated normally.
 
-Cache-only mode uses the settings provided by the current process. Keep these values exactly the same, including set versus unset: `GGML_OPENVINO_REQUANT_KQUANT`, `GGML_OPENVINO_NATIVE_SOFTPLUS`, `GGML_OPENVINO_DISABLE_KV_SLICE`, `GGML_OPENVINO_MANUAL_GQA_ATTN`, `GGML_OPENVINO_STATEFUL_EXECUTION`, `GGML_OPENVINO_DISABLE_KV_STATE_RELAYOUT`, `GGML_OPENVINO_DISABLE_REMOTE_OUTPUTS`, `GGML_OPENVINO_REDUCE_COMPILE_MEM`, `GGML_OPENVINO_MEMORY_OPTIMIZE`, `GGML_OPENVINO_PROFILING`, `GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK`, and `GGML_OPENVINO_DEBUG_NODE`. On GPU, also repeat `GGML_OPENVINO_MOE_OP=0` if used. `GGML_OPENVINO_SPILL_DIR` is optional on the first run and ignored in cache-only mode; host-weight release is disabled in cache-only mode.
+Cache-only mode uses the settings provided by the current process. Keep these values exactly the same, including set versus unset: `GGML_OPENVINO_REQUANT_KQUANT`, `GGML_OPENVINO_NATIVE_SOFTPLUS`, `GGML_OPENVINO_KV_SLICE`, `GGML_OPENVINO_MANUAL_GQA_ATTN`, `GGML_OPENVINO_STATEFUL_EXECUTION`, `GGML_OPENVINO_DISABLE_KV_STATE_RELAYOUT`, `GGML_OPENVINO_DISABLE_REMOTE_OUTPUTS`, `GGML_OPENVINO_REDUCE_COMPILE_MEM`, `GGML_OPENVINO_MEMORY_OPTIMIZE`, `GGML_OPENVINO_PROFILING`, `GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK`, and `GGML_OPENVINO_DEBUG_NODE`. On GPU, also repeat `GGML_OPENVINO_MOE_OP=0` if used. `GGML_OPENVINO_SPILL_DIR` is optional on the first run and ignored in cache-only mode; host-weight release is disabled in cache-only mode.
 
 A missing or incompatible graph fails with an error instead of compiling with absent weights. The fingerprint uses the dynamic graph's topology, ports, model parameters, weights, settings, and OpenVINO version; changing only dynamic token or KV sizes does not require a new entry. A different workload can still require another graph; populate it first without cache-only mode.
 

ggml/src/ggml-openvino/ggml-decoder.cpp

@@ -1036,7 +1036,7 @@ std::pair<ModelParams, ComputeParams> GgmlOvDecoder::compute_llm_params(ggml_cgr
                 compute_params.mask_name = mask_name;
             }
             if (is_static) {
-                const bool slice_prefill = ggml_openvino_static_kv_slice_enabled() && compute_params.input_len > 1;
+                const bool slice_prefill = ggml_openvino_kv_slice_enabled() && compute_params.input_len > 1;
                 if (!slice_prefill) {
                     compute_params.attention_size = model_params.ctx_per_seq;
                 }
@@ -1059,7 +1059,7 @@ std::pair<ModelParams, ComputeParams> GgmlOvDecoder::compute_llm_params(ggml_cgr
             node->src[0]->src[0]->op == GGML_OP_VIEW) {
             compute_params.attention_size = node->ne[0];
             compute_params.mask_name.clear();
-            if (is_static && !(ggml_openvino_static_kv_slice_enabled() && compute_params.input_len > 1)) {
+            if (is_static && !(ggml_openvino_kv_slice_enabled() && compute_params.input_len > 1)) {
                 compute_params.attention_size = model_params.ctx_per_seq;
             }
         }
@@ -1215,7 +1215,7 @@ ov::PartialShape GgmlOvDecoder::get_graph_input_shape(const ggml_tensor * op,
             // Both masks are named "attn_inp_kq_mask", so tell them apart by tensor identity.
             const bool swa_mask = is_swa_mask(input);
             int attention_size = swa_mask ? m_model_params.ctx_per_seq_swa * m_model_params.n_seq : m_model_params.ctx;
-            if (m_is_prefill && ggml_openvino_static_kv_slice_enabled()) {
+            if (m_is_prefill && ggml_openvino_kv_slice_enabled()) {
                 attention_size = swa_mask ? m_compute_params.attention_size_swa : m_compute_params.attention_size;
             }
             input_shape = ov::PartialShape{1, 1, m_is_prefill ? m_prefill_chunk_size : 1, attention_size};
@@ -1241,7 +1241,7 @@ ov::PartialShape GgmlOvDecoder::get_graph_input_shape(const ggml_tensor * op,
         if (!m_is_static && !is_flat_kv) {
             // do not fix ctx size to make llama-bench work across test params
             input_shape[2] = -1;
-        } else if (m_is_prefill && ggml_openvino_static_kv_slice_enabled()) {
+        } else if (m_is_prefill && ggml_openvino_kv_slice_enabled()) {
             const auto layer = extract_layer_from_name(name);
             if (layer.has_value() && !is_swa_layer(*layer) && m_compute_params.attention_size > 0 &&
                 m_compute_params.attention_size < input_shape[2].get_length()) {

ggml/src/ggml-openvino/ggml-openvino-extra.cpp

@@ -144,7 +144,6 @@ void ggml_openvino_device_config::init() {
         "GGML_OPENVINO_PRINT_CGRAPH_TENSOR_ADDRESS",
         "GGML_OPENVINO_ENABLE_CACHE",
         "GGML_OPENVINO_DISABLE_CACHE",
-        "GGML_OPENVINO_DISABLE_KV_SLICE",
         "GGML_OPENVINO_ENABLE_KV_4D",
         "GGML_OPENVINO_ENABLE_FALLBACK",
         "GGML_OPENVINO_KV_SCATTER_ELEMENTS",
@@ -166,7 +165,6 @@ void ggml_openvino_device_config::init() {
         "GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK",
         "GGML_OPENVINO_GDN_Q_SCALE",
         "GGML_OPENVINO_NPU_L0_HOST_TENSORS",
-        "GGML_OPENVINO_NPU_KV_SLICE",
         "GGML_OPENVINO_KV_SLICE",
         "GGML_OPENVINO_NPU_HFA_PREFILL",
         "GGML_OPENVINO_NPU_DECODE_UNFOLD",
@@ -351,29 +349,8 @@ bool ggml_openvino_is_gpu() {
     return has_prefix(ggml_openvino_get_device_name(), "GPU");
 }
 
-// -1: GGML_OPENVINO_KV_SLICE unset, use the mode's default (or its deprecated alias)
-static int kv_slice_setting() {
-    const char * value = ggml_openvino_getenv_str("GGML_OPENVINO_KV_SLICE");
-    if (value == nullptr || *value == '\0') {
-        return -1;
-    }
-    return ggml_openvino_getenv_int("GGML_OPENVINO_KV_SLICE") != 0 ? 1 : 0;
-}
-
-bool ggml_openvino_dynamic_kv_slice_enabled() {
-    const int setting = kv_slice_setting();
-    if (setting >= 0) {
-        return setting != 0;
-    }
-    return ggml_openvino_getenv_int("GGML_OPENVINO_DISABLE_KV_SLICE") == 0;
-}
-
-bool ggml_openvino_static_kv_slice_enabled() {
-    const int setting = kv_slice_setting();
-    if (setting >= 0) {
-        return setting != 0;
-    }
-    return ggml_openvino_getenv_int("GGML_OPENVINO_NPU_KV_SLICE") != 0;
+bool ggml_openvino_kv_slice_enabled() {
+    return ggml_openvino_getenv_int("GGML_OPENVINO_KV_SLICE", 1) != 0;
 }
 
 bool ggml_openvino_npu_fast_mask_enabled() {

ggml/src/ggml-openvino/ggml-openvino-extra.h

@@ -117,17 +117,16 @@ bool ggml_openvino_release_weights_enabled();
 // Check if running on NPU
 bool ggml_openvino_is_npu();
 
-// NPU-only opt-in performance toggles (all default off). Centralized here so each
-// env-var name lives in one place; see docs/backend/OPENVINO.md for semantics.
 // KV-cache slicing: bind the KV cache, and size the KQ masks, at the attended length instead of the full context.
 // One mechanism, two modes:
-//   dynamic graphs (CPU/GPU): the KV input is a view over the cache; the graph is not recompiled. On by default.
+//   dynamic graphs (CPU/GPU): the KV input is a view over the cache; the graph is not recompiled.
 //   static graphs (NPU, or any device with GGML_OPENVINO_FORCE_STATIC): the prefill graph is compiled at the aligned
-//   attended length and recompiled when it changes; decode keeps the full context. Off by default.
-// GGML_OPENVINO_KV_SLICE: unset = the default of each mode, 0 = off in both, 1 = on in both.
-// Deprecated aliases: GGML_OPENVINO_DISABLE_KV_SLICE=1 (dynamic off), GGML_OPENVINO_NPU_KV_SLICE=1 (static on).
-bool ggml_openvino_dynamic_kv_slice_enabled();
-bool ggml_openvino_static_kv_slice_enabled();
+//   attended length and recompiled when it changes; decode keeps the full context.
+// On by default for all devices; GGML_OPENVINO_KV_SLICE=0 turns it off.
+bool ggml_openvino_kv_slice_enabled();
+
+// NPU-only opt-in performance toggles (all default off). Centralized here so each
+// env-var name lives in one place; see docs/backend/OPENVINO.md for semantics.
 bool ggml_openvino_npu_fast_mask_enabled();
 bool ggml_openvino_npu_l0_host_tensors_enabled();
 bool ggml_openvino_npu_hfa_prefill_enabled();

ggml/src/ggml-openvino/model-cache.cpp

@@ -173,7 +173,6 @@ bool ggml_openvino_model_cache_only() {
 static const char * cache_settings[] = {
     "GGML_OPENVINO_REQUANT_KQUANT",
     "GGML_OPENVINO_NATIVE_SOFTPLUS",
-    "GGML_OPENVINO_DISABLE_KV_SLICE",
     "GGML_OPENVINO_KV_SLICE",
     "GGML_OPENVINO_MANUAL_GQA_ATTN",
     "GGML_OPENVINO_STATEFUL_EXECUTION",

ggml/src/ggml-openvino/utils.cpp

@@ -57,11 +57,9 @@ namespace {
 std::optional<ov::Tensor> try_make_kv_sliced_tensor(const std::shared_ptr<GgmlOvDecoder> & ggml_decoder,
                                                     const std::string & name,
                                                     const ggml_tensor * ggml_tensor) {
-    // Static graphs bake the sliced length into the prefill graph (see ggml_openvino_static_kv_slice_enabled);
-    // decode keeps its full-context graph.
-    const bool slice = ggml_decoder->is_static() ?
-                           ggml_decoder->m_is_prefill && ggml_openvino_static_kv_slice_enabled() :
-                           ggml_openvino_dynamic_kv_slice_enabled();
+    // Static graphs bake the sliced length into the prefill graph; decode keeps its full-context graph.
+    const bool slice =
+        ggml_openvino_kv_slice_enabled() && (!ggml_decoder->is_static() || ggml_decoder->m_is_prefill);
     if (!slice || ggml_decoder->is_stateful()) {
         return std::nullopt;
     }
@@ -142,7 +140,7 @@ std::optional<ov::Tensor> try_make_kv_4d_tensor(const std::shared_ptr<GgmlOvDeco
 // Column count of a static KQ mask input; must match GgmlOvDecoder::get_graph_input_shape().
 static size_t get_static_attention_size(std::shared_ptr<GgmlOvDecoder> ggml_decoder, const std::string & name) {
     const bool is_swa = ggml_decoder->is_swa_mask(ggml_decoder->get_input_ggml_tensor(name));
-    if (ggml_decoder->m_is_prefill && ggml_openvino_static_kv_slice_enabled()) {
+    if (ggml_decoder->m_is_prefill && ggml_openvino_kv_slice_enabled()) {
         const auto params = ggml_decoder->get_compute_params();
         const int attention_size = is_swa ? params.attention_size_swa : params.attention_size;
         if (attention_size > 0) {
@@ -165,7 +163,7 @@ static uint64_t ggml_openvino_model_cache_extra_cfg(bool stateful, bool recurren
     uint64_t extra_cfg = 1;  // Graph-ordinal port names (invalidate older disk-cache blobs).
     extra_cfg = extra_cfg * 131 + (stateful ? (recurrent ? 2u : 1u) : 0u);
     extra_cfg = extra_cfg * 131 + (ggml_openvino_reduce_compile_mem_enabled() ? 1u : 0u);
-    extra_cfg = extra_cfg * 131 + (ggml_openvino_dynamic_kv_slice_enabled() ? 0u : 1u);
+    extra_cfg = extra_cfg * 131 + (ggml_openvino_kv_slice_enabled() ? 0u : 1u);
     extra_cfg = extra_cfg * 131 + (ggml_openvino_getenv_int("GGML_OPENVINO_ENABLE_KV_4D") > 0 ? 1u : 0u);
     extra_cfg = extra_cfg * 131 + (manual_gqa_enabled ? 1u : 0u);
     extra_cfg = extra_cfg * 131 + (ggml_openvino_getenv_int("GGML_OPENVINO_PROFILING") >= 2 ? 1u : 0u);
@@ -1736,7 +1734,7 @@ enum ggml_status ov_graph_compute_static(ggml_cgraph * cgraph, const std::shared
         ggml_decoder = entry->ptr;
         old_m_params = ggml_decoder->get_model_params();
         cache_hit = old_m_params.can_reuse_statically(m_params);
-        if (cache_hit && is_prefill && ggml_openvino_static_kv_slice_enabled()) {
+        if (cache_hit && is_prefill && ggml_openvino_kv_slice_enabled()) {
             const auto old_c_params = ggml_decoder->get_compute_params();
             cache_hit = old_c_params.attention_size == c_params.attention_size &&
                         old_c_params.attention_size_swa == c_params.attention_size_swa;
@@ -1814,7 +1812,7 @@ enum ggml_status ov_graph_compute_static(ggml_cgraph * cgraph, const std::shared
             GGML_LOG_DEBUG("ggml-openvino: shared compiled model HIT (static)\n");
         } else {
             auto decode_c_params = c_params;
-            if (ggml_openvino_static_kv_slice_enabled()) {
+            if (ggml_openvino_kv_slice_enabled()) {
                 decode_c_params.attention_size = m_params.ctx_per_seq;
                 decode_c_params.attention_size_swa = m_params.ctx_per_seq_swa;
             }
@@ -1841,7 +1839,7 @@ enum ggml_status ov_graph_compute_static(ggml_cgraph * cgraph, const std::shared
                 base_extra = ggml_openvino_model_cache_mix_string(
                     base_extra, ggml_openvino_getenv_str("GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT"));
                 base_extra = base_extra * 131 + static_cast<uint64_t>(ggml_openvino_getenv_int("GGML_OPENVINO_NPU_KEEP_Q4_0"));
-                base_extra = base_extra * 131 + (ggml_openvino_static_kv_slice_enabled() ? 1u : 0u);
+                base_extra = base_extra * 131 + (ggml_openvino_kv_slice_enabled() ? 1u : 0u);
                 base_extra = ggml_openvino_model_cache_mix_string(
                     base_extra, ggml_openvino_getenv_str("GGML_OPENVINO_NPU_CONFIG"));
                 base_extra = ggml_openvino_model_cache_mix_string(