vLLM 0.19.0 + ROCm 7.12: Triton HIP invalid device ordinal on gfx1151 (Strix Halo)

Environment

• GPU: AMD Radeon 8060S (Strix Halo, gfx1151)
• Package: vllm0.19.0-rocm7.12.0-gfx1151-x64 (portable build from this repo)
• Kernel: Linux 7.0.0-rc7 (CachyOS)
• RAM: 128GB unified
• PyTorch HIP: 7.12.60610
• Triton: 3.6.0

Bug

vLLM fails to start with any GGUF model. The EngineCore subprocess crashes during attention initialization with:

RuntimeError: Triton Error [HIP]: Code: 101, Messsage: invalid device ordinal

The error comes from Triton's AMD HIP driver (triton/backends/amd/driver.py) when ROCM_ATTN executes Triton kernels in the forked EngineCore child process.

Key observation

PyTorch HIP works fine in the main process:

import torch
print(torch.cuda.device_count())   # 1
print(torch.cuda.get_device_name(0))  # Radeon 8060S Graphics
x = torch.randn(100, 100, device="cuda:0", dtype=torch.float16)
y = torch.matmul(x, x)  # OK

The crash only happens in the EngineCore subprocess (pid different from APIServer). This suggests Triton's HIP device enumeration breaks after fork/spawn on gfx1151 iGPU.

Reproduction

export PYTHONPATH="$HOME/vllm-rocm/lib/python3.12/site-packages/_rocm_sdk_core/share/amd_smi:$PYTHONPATH"
export HIP_VISIBLE_DEVICES=0

~/vllm-rocm/bin/python3 -m vllm.entrypoints.openai.api_server \
  --model any-model.gguf \
  --port 8000 \
  --max-model-len 4096 \
  --dtype float16 \
  --gpu-memory-utilization 0.70 \
  --enforce-eager

Note: amdsmi also needed a PYTHONPATH fix — it lives at _rocm_sdk_core/share/amd_smi/ but isn't importable by default. Without it, vLLM can't detect the ROCm platform at all.

What I tried (all fail with same Triton error)

• --enforce-eager (skips inductor, but ROCM_ATTN still uses Triton)
• HIP_VISIBLE_DEVICES=0
• VLLM_USE_TRITON_FLASH_ATTN=0
• VLLM_USE_V1=0
• --disable-custom-all-reduce
• VLLM_WORKER_MULTIPROC_METHOD=spawn

Stack trace

(EngineCore pid=632069) ERROR [core.py:1108]
  File "triton/backends/amd/driver.py", line 835, in __call__
    self.launch(self.launch_cooperative_grid, gridX, gridY, gridZ, stream, function, profile_scratch, *args)
  File "triton/knobs.py", line 423, in __call__
    for call in self.calls if not self.reversed else reversed(self.calls):
SystemError: <built-in method __reversed__ of list object at 0x7fe8f669ea00> returned a result with an exception set
RuntimeError: Triton Error [HIP]: Code: 101, Messsage: invalid device ordinal

Expected

vLLM should be able to serve GGUF models on gfx1151 using the portable ROCm 7.12 package, same as it does on discrete GPUs.

Impact

gfx1151 (Strix Halo) has 128GB unified VRAM — it's an ideal target for running large models locally. The portable package detects the GPU correctly and loads model weights fine, but inference fails at the Triton kernel launch stage.

Currently using llama.cpp Vulkan as a workaround via Lemonade SDK, which works at ~80 tok/s on 35B MoE models.

Update: RESOLVED via lemonade-sdk/lemonade#1537

The Triton HIP invalid device ordinal crash is solved when vLLM is launched through Lemonade's wrapped server backend (PR #1537 by @ramkrishna2910).

What fixed it

PR #1537 adds vLLM as a WrappedServer backend in Lemonade. The key difference from a raw python -m vllm.entrypoints.openai.api_server launch is that Lemonade's ProcessManager sets up the environment correctly (PYTHONNOUSERSITE=1, FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE, proper LD_LIBRARY_PATH via the vllm-server launcher script). The Triton subprocess device enumeration works correctly through this path.

Test results — vLLM ROCm on gfx1151 (Strix Halo)

Built from the test-vllm branch, ran all 14 catalog models + a 72B monster:

Model                          Params    Mean tok/s    ±StdDev
Qwen3-0.6B                    0.6B       116.7         ±0.2
Llama-3.2-1B-Instruct (AWQ)   1B         110.4         ±0.0
Llama-3.2-3B-Instruct (AWQ)   3B          50.5         ±0.1
Qwen3.5-2B                    2B          44.0         ±0.0
Qwen3-4B-AWQ                  4B          42.8         ±0.0
Qwen3-4B                      4B          25.4         ±0.0
Phi-4-mini                    3.8B        25.1         ±0.0
Qwen3-8B-AWQ                  8B          22.8         ±0.0
Qwen3-8B                      8B          12.3         ±0.0
Qwen3.5-9B                    9B          11.6         ±0.0
Qwen2.5-72B-Instruct-AWQ      72B          2.3         ±0.1

12/14 catalog models passed. 72B dense on 128GB unified memory works.

Environment

• Hardware: AMD Strix Halo, 128GB unified, gfx1151
• OS: CachyOS (Arch), kernel 7.0.0-1-mainline
• vllm-rocm: 0.19.0 + ROCm 7.12.0 (installed via Lemonade /v1/install)

Root cause note

The raw vLLM launch (outside Lemonade) still crashes with the Triton HIP error. The issue is specifically in how the EngineCore subprocess inherits the HIP device environment. Lemonade's wrapper handles this correctly. This may still be worth investigating for standalone vLLM users on gfx1151.