Repository navigation
docs(p5en): add the NCCL NVLS fix that blocks every multi-GPU serve - #173
Merged
Merged
Conversation
Serving any TP=8 model on a p5en.48xlarge failed at engine init with "NCCL error: unhandled cuda error" and "Engine core initialization failed", before any weights loaded. The real cause only appears under NCCL_DEBUG=INFO: NCCL cannot bind NVLink SHARP (NVLS) multicast memory and the driver returns CUDA 401, which NCCL treats as fatal. NCCL names the switch itself. Setting NCCL_NVLS_ENABLE=0 takes a bare 8-GPU all_reduce from every rank failing to world=8. Record it as Fix 4, and rewrite the section preamble, which claimed two failure points and described them all as CUDA JIT problems. Fix 4 is neither: it fails at distributed init rather than at kernel link time, and it applies to every multi-GPU model regardless of precision. Three things cost time here and are now written down: - A clean fabric does not rule this out. Fabric Manager was active and reported Completed/Success on all 8 GPUs, with NVLink at 26.562 GB/s, while the multicast bind still failed. - The aws-ofi-nccl warnings printed alongside it are unrelated; NCCL_NET_PLUGIN=none does not help, and a single-node TP job never uses the network plugin. - Disabling NVLS drops in-network reduction, so TP=8 collectives are slower. Cost per task derived from wall clock on such a run is not comparable to a run with NVLS working, and results docs should say so. Also add the bare torch.distributed all_reduce that reproduces it without vLLM, which separates a node fault from a serving bug in about a minute.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Serving any TP=8 model on a
p5en.48xlargefailed at engine init, before any weights loaded:Neither line names the real cause. Under
NCCL_DEBUG=INFOit appears:The existing note covers three CUDA JIT failures on this node but not this one, which is the first thing a multi-GPU serve hits and the only one that fails before weight loading.
What changed
.claude/skills/vllm-setup/p5en-h200-cuda-fixes.mdgains Fix 4 (NCCL_NVLS_ENABLE=0), a row in the quick failure reference, the export in the combined launch block, and a correctedModel -> which fixesline: Fix 4 applies to every multi-GPU model whatever its precision, because it fails before a model is loaded.The section preamble is rewritten. It claimed startup fails "in two places" and described all of them as CUDA JIT problems; there are now four, and Fix 4 is neither a JIT nor a link-time failure.
Three things that cost time, now recorded
nvidia-smi -qreportedState: Completed / Status: Successon all 8 GPUs,nvidia-fabricmanagerwas active, and NVLink showed 26.562 GB/s per link. Section 0 was already satisfied. The multicast bind still failed.aws-ofi-ncclwarnings alongside it are a red herring.NCCL_NET_PLUGIN=nonedoes not fix it, and a single-node TP job never uses the network plugin.The note also adds the bare 8-GPU
torch.distributedall_reduce that reproduces the failure with no vLLM involved, which separates a node fault from a serving bug in about a minute.Verification
Measured on a
p5en.48xlarge: driver 595.91.07, NCCL 2.29.7, torch 2.13.0+cu130, vLLM 0.28.0.ncclUnhandledCudaError, both inside vLLM and in the bare all_reduce.NCCL ALL_REDUCE OK, world=8, sum=8.0, anddeepseek-v3.2serves at TP=8 with a 131072 window, through KV-cache init and CUDA graph capture.Documentation only.