Repository navigation
Conversation
On layers without RoPE (SmolLM3: every 4th layer), Q reaches FLASH_ATTN_EXT through PERMUTE -> VIEW -> RESHAPE. The support check rejected this pattern, causing these layers to fall back to CPU and the split graph to fail with -fa on/auto.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
SmolLM3 skips RoPE on every 4th layer. On these layers, Q reaches
FLASH_ATTN_EXTthrough the following chain:PERMUTE -> VIEW -> RESHAPEThis is the stream view created in
build_attn_mhaover thereshape_3doutput of the Q matmul.is_supported_flash_attn_patterndid not recognize this pattern, so with-fa on/auto, these 9 layers fell back to CPU. This caused the graph to split and the GPU portion to fail with 4 different errors across stateful and stateless execution.This PR extends the support check to accept:
PERMUTE -> VIEW -> RESHAPE -> baseThe existing translation already handles this Q path:
VIEWis a no-op on the dynamic path, and-fa offuses the same Q representation.Additional information
Tested on Arc Pro B70 with OpenVINO 2026.4.1 using
dev_backend_openvinocommitb1d8ce791and the ticket command (1024-token prompt,-c 1382 -n 128).Decode performance
-fa on-fa off-fa autoValues are decode tok/s, shown as before -> after.
Accuracy
KLD against the CPU backend, measured on 4 WikiText-2 slices with context length 1024:
-fa on0.0435, same as-fa off0.0435-fa on0.0491 vs-fa off0.0441-fa off0.0438, stateless-fa off0.0435Additional validation
25e3135cc+ ggml-openvino: keep stateful execution with -fa off #361 + ggml-openvino: support gemma-2 soft-capped attention #362):-fa on/autonow pass in both stateful and stateless modes.25e3135cc+ ggml-openvino: keep stateful execution with -fa off #361 + ggml-openvino: support gemma-2 soft-capped attention #362, this change applied together with the Ling-mini fixes): 7 other models × 6 configurations, identical greedy output in all 42 pairs.Requirements