Skip to content

ggml-openvino: accept flash attention on layers without RoPE - #372

Open
osabnis wants to merge 1 commit into
ravi9:dev_backend_openvinofrom
osabnis:ov-smollm3-fa-nope
Open

osabnis wants to merge 1 commit into
ravi9:dev_backend_openvinofrom
osabnis:ov-smollm3-fa-nope

Conversation

@osabnis

@osabnis osabnis commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator

Overview

SmolLM3 skips RoPE on every 4th layer. On these layers, Q reaches FLASH_ATTN_EXT through the following chain:

PERMUTE -> VIEW -> RESHAPE

This is the stream view created in build_attn_mha over the reshape_3d output of the Q matmul.

is_supported_flash_attn_pattern did not recognize this pattern, so with -fa on/auto, these 9 layers fell back to CPU. This caused the graph to split and the GPU portion to fail with 4 different errors across stateful and stateless execution.

This PR extends the support check to accept:

PERMUTE -> VIEW -> RESHAPE -> base

The existing translation already handles this Q path: VIEW is a no-op on the dynamic path, and -fa off uses the same Q representation.

Additional information

Tested on Arc Pro B70 with OpenVINO 2026.4.1 using dev_backend_openvino commit b1d8ce791 and the ticket command (1024-token prompt, -c 1382 -n 128).

Decode performance

Execution -fa on -fa off -fa auto
Stateful fail -> 130.0 131.4 -> 134.9 fail -> 128.1
Stateless fail -> 106.0 98.1 -> 98.5 fail -> 106.4

Values are decode tok/s, shown as before -> after.

Accuracy

KLD against the CPU backend, measured on 4 WikiText-2 slices with context length 1024:

  • Stateless: -fa on 0.0435, same as -fa off 0.0435
  • Stateful: -fa on 0.0491 vs -fa off 0.0441
  • Unpatched control: stateful -fa off 0.0438, stateless -fa off 0.0435

Additional validation

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - AI assistance was used for analysis, editing the PR description, and running tests.

On layers without RoPE (SmolLM3: every 4th layer), Q reaches
FLASH_ATTN_EXT through PERMUTE -> VIEW -> RESHAPE. The support check
rejected this pattern, causing these layers to fall back to CPU and the
split graph to fail with -fa on/auto.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant