ROCm build of llama.cpp is suboptimal for Strix Halo

[pwilkin]

Platform

Linux/Ubuntu

Lemonade Version

10.9.0

GPU / APU Model

Strix Halo [Radeon Graphics / Radeon 8050S Graphics / Radeon 8060S Graphics]

Component

llama.cpp

Bug Description

The ROCm build for llama.cpp ships with -DGGML_HIP_ROCWMMA_FATTN=ON, which absolutely MURDERS long-context prefill on Strix Halo.

Steps to Reproduce

(venv) ilintar@piotr-strixhalo:/devel/tools/llama.cpp$ llama-bench -dev ROCM0 -d 0,30000 -hf ilintar/Ornith-1.0-35B-GGUF:IQ4_XS

ROCWMMA_FATTN = OFF

model	size	params	backend	threads	dev	test	t/s
qwen35moe 35B.A3B Q4_K - Medium	17.57 GiB	34.66 B	ROCm,Vulkan,BLAS	16	ROCm0	pp512	1152.00 ± 30.86
qwen35moe 35B.A3B Q4_K - Medium	17.57 GiB	34.66 B	ROCm,Vulkan,BLAS	16	ROCm0	tg128	63.45 ± 0.45
qwen35moe 35B.A3B Q4_K - Medium	17.57 GiB	34.66 B	ROCm,Vulkan,BLAS	16	ROCm0	pp512 @ d30000	626.35 ± 5.89
qwen35moe 35B.A3B Q4_K - Medium	17.57 GiB	34.66 B	ROCm,Vulkan,BLAS	16	ROCm0	tg128 @ d30000	54.56 ± 0.23

ROCWMMA_FATTN = ON

model	size	params	backend	threads	dev	test	t/s
qwen35moe 35B.A3B Q4_K - Medium	17.57 GiB	34.66 B	ROCm,Vulkan,BLAS	16	ROCm0	pp512	1081.16 ± 9.02
qwen35moe 35B.A3B Q4_K - Medium	17.57 GiB	34.66 B	ROCm,Vulkan,BLAS	16	ROCm0	tg128	63.23 ± 0.31
qwen35moe 35B.A3B Q4_K - Medium	17.57 GiB	34.66 B	ROCm,Vulkan,BLAS	16	ROCm0	pp512 @ d30000	143.91 ± 0.65
qwen35moe 35B.A3B Q4_K - Medium	17.57 GiB	34.66 B	ROCm,Vulkan,BLAS	16	ROCm0	tg128 @ d30000	42.18 ± 0.19

Expected vs Actual Behavior

Expected: ship both variants, add a switch for the variant and switch according to GPU
Actual: only the ON variant is shipped, resulting in terrible long-context prefill performance

[hiSandog]

Runtime selection by detected GPU architecture would avoid making every user choose a build flag, but the packaged variants need a clear compatibility boundary so caches/models remain shared and only the backend library differs. The benchmark gate should include short and 30k-context prefill plus token generation on Strix Halo and at least one discrete ROCm GPU where ROCWMMA is beneficial. That would justify an architecture allow/deny list and catch upstream llama.cpp changes that alter which variant wins.

[ianbmacdonald]

@lhl - is this your ROCm regression?