vllm-server launcher does not set PYTHONNOUSERSITE=1 — ~/.local can shadow bundled torch/triton/amdsmi

[liminfei-amd]

Summary

The generated bin/vllm-server launcher (in .github/workflows/build-vllm-rocm.yml) sets LD_LIBRARY_PATH, PYTHONPATH (amd_smi) and FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE, but does not set PYTHONNOUSERSITE=1.

The bundled interpreter is a relocatable python-build-standalone CPython (not a venv), so site.ENABLE_USER_SITE = True. That puts ~/.local/lib/python3.12/site-packages on sys.path ahead of the bundle's own site-packages, and a user who previously ran pip install --user torch / triton / amdsmi gets that copy loaded instead of the bundled, ROCm-matched one. Setting PYTHONNOUSERSITE=1 in the launcher makes the bundled packages always win.

Evidence (measured on the real vllm0.22.1-rocm7.13.0-gfx1151 bundle)

Bundle interpreter: CPython 3.12.11; bundled torch 2.11.0+rocm7.13, triton 3.6.0. site.ENABLE_USER_SITE = True; ~/.local precedes the bundle on sys.path.

Installed a stock (CUDA-built) triton==3.1.0 into ~/.local, then ran the bundle interpreter with the env the launcher sets:

~/.local state	import triton resolves to
stock CUDA triton 3.1.0 present, launcher as-is	~/.local CUDA triton 3.1.0 (wrong)
same, with PYTHONNOUSERSITE=1	bundled triton 3.6.0 (ROCm 7.13)

So as shipped, the process can end up running a CUDA-built triton against the bundle's ROCm torch. PYTHONNOUSERSITE=1 makes both come from the bundle.

Scope (honest)

This is a robustness hardening for the launcher. It is not a verified fix for the HIP 101 invalid device ordinal report (#3 / triton-lang/triton#10036): when I drove this end-to-end (working around the missing clang-23 by pointing Triton's JIT at a system ROCm clang), a simple Triton kernel still ran fine in a spawn child even with the CUDA-built triton shadowing the bundle — no HIP 101. So I cannot attribute #3 to this; reproducing that crash likely needs the reporter's exact conflicting version and/or vLLM's actual attention kernels (ROCM_ATTN/TRITON_ATTN), which I did not exercise.

Using bundled packages unconditionally is still the right behavior for a self-contained bundle, independent of #3.

Proposed change (1 line)

In the vllm-server launcher heredoc, before the final exec:

export PYTHONNOUSERSITE=1   # ignore ~/.local so the bundled torch/triton/amdsmi are used

Draft PR: #15.

---

Separate, unrelated finding while testing: this bundle is missing bin/clang-23. The 26 KB clang driver re-execs clang-23, which is absent (the LLVM 23 runtime files under lib/clang/23/ are present, but the main clang-23 binary is not), so Triton's JIT cannot compile with the bundled toolchain unless a system clang is supplied. Happy to file that separately if useful.

AI-assisted investigation; findings verified on the real bundle.

[liminfei-amd]

Opened PR #15 with the 1-line launcher change (robustness; see the issue's Scope note — not claimed as a #10036 fix).