wheels: RAPIDS is not installable alongside PyTorch 2.12 CUDA 12 wheels

[jameslamb]

Description

RAPIDS 26.06 can't be installed alongside the latest PyTorch CUDA 12 wheels.

In its 2.12 release (May 13, 2026), PyTorch dropped its CUDA 12.8 and 12.9 builds:

• Remove cuda 12.9 cd builds pytorch/pytorch#179155
• [BE] Remove cuda 12.9 from docker. Use linux.arm64.4xlarge for Linux aarch64 cuda builds pytorch/pytorch#179158
• [RFC] CUDA support matrix for Release 2.12 - introduce CUDA 13.2 as experimental. pytorch/pytorch#178665

Those 12.9 wheels were the only torch CUDA 12 wheels RAPIDS could be installed alongside without workarounds like --no-deps.

Details

The only CUDA 12 build available for PyTorch 2.12 is for CUDA 12.6.

Its == pinned to the nvidia-nvjitlink-cu12 from CUDA 12.6, and so is incompatible with the nvidia-nvjitlink-cu12>=12.9 requirement carried by several RAPIDS libraries.

$ docker run --rm -it python:3.12 bash

❌ CUDA 12.6 + PyTorch 2.12 + RAPIDS 26.06

$ INDEX_ARGS=(--extra-index-url=https://download.pytorch.org/whl/cu126 --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple/)
$ pip install "${INDEX_ARGS[@]}" 'cuvs-cu12==26.6.*,>=26.6.0a0' 'torch==2.12.0+cu126'
ERROR: Cannot install cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.6.3 and cuvs-cu12 because these package versions have conflicting dependencies.

The conflict is caused by:
    libcuvs-cu12 26.6.0a115 depends on nvidia-nvjitlink-cu12<13 and >=12.9
    cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx] 12.6.3 depends on nvidia-nvjitlink-cu12==12.6.85.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "nvjitlink"

Installation in CUDA 13 environments is fine.

✅ CUDA 13.0 + PyTorch 2.12 + RAPIDS 26.06

$ INDEX_ARGS=(--extra-index-url=https://download.pytorch.org/whl/cu130 --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple/)
$ pip install "${INDEX_ARGS[@]}" 'cuvs-cu13==26.6.*,>=26.6.0a0' 'torch==2.12.0+cu130'
Successfully installed ... cuda-toolkit-13.0.2 cuvs-cu13-26.6.0 ... nvidia-nvjitlink-13.0.88 ... torch-2.12.0+cu130 ...

✅ CUDA 13.2 + PyTorch 2.12 + RAPIDS 26.06

$ INDEX_ARGS=(--extra-index-url=https://download.pytorch.org/whl/cu132 --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple/)
$ pip install "${INDEX_ARGS[@]}" 'cuvs-cu13==26.6.*,>=26.6.0a0' 'torch==2.12.0+cu132'
Successfully installed ... cuda-toolkit-13.2.1 cuvs-cu13-26.6.0 ... nvidia-nvjitlink-13.3.33 ... torch-2.12.0+cu132 ...

Opening this to track work towards resolving this for future torch and RAPIDS versions.

Benefits of this work

• resolving this would allow use of the latest RAPIDS and latest torch together in CUDA 12 environments

Approach

RAPIDS strongly prefers to build its CUDA 12 wheels with the v12.9 of the CUDA toolkit, and so needs that runtime dependency of nvidia-nvjitlink-cu12>=12.9.

Other options that could resolve this (in no particular order):

• recommending that RAPIDS users pip install --no-deps PyTorch wheels in CUDA 12 environments
• RAPIDS loosening its nvidia-nvjitlink-cu12 requirement
  • ideas like building against an older CTK, statically linking libnvJitLink, and runtime shim tricks have been tried before and have various technical limitations, but including this option for completeness
• torch restoring CUDA 12.9 builds
• torch loosening its nvidia-nvjitlink-cu12 requirement for CUDA 12 wheels (to something like >=12.6,<13)
• cuda-toolkit==12.6.* metapackage loosening its nvidia-nvjitlink-cu12 requirement
  • as was done in 12.8.2, 12.9.2, 13.0.3, 13.1.2, and from 13.2.1 onwards

Task List

• loosen torch CUDA 12.x wheels' nvidia-nvjitlink pin (CUDA 12 wheels: relax nvidia-nvjitlink-cu12 runtime dependency pytorch/pytorch#186958)
• wait for a release of torch that has CUDA 12.x wheels with that newer, relaxed pin
  • likely 2.13.1 or 2.14.0, see wheels: RAPIDS is not installable alongside PyTorch 2.12 CUDA 12 wheels #289 (comment)
• expand PyTorch testing to use torch=={v}+cu126 for cugraph-gnn and any other affected RAPIDS (see build CUDA 13 wheels with latest CTK (13.3.0), require C++20 for all builds cugraph-gnn#487 for context)

Notes

It's possible this will just be resolved by torch restoring CUDA 12.9 builds, see:

• request for PyTorch 2.12 cu129 binary pytorch/pytorch#185269
• Revive CUDA 12.9 nightly binary builds pytorch/pytorch#186015

[jameslamb]

I'd hoped we'd be able to close this once torch 2.13.0 was released (planned for tomorrow, July 8), but it looks to me like pytorch/pytorch#186958 did not make it into 2.13. That commit is missing from the release/2.13 branch on pytorch/pytorch, and the release candidate wheels for 2.13.0 still depend on cuda-toolkit[..., nvjitlink, ...]==12.6.3 😫

That PR was merged June 12th and the 2.13 release branch was cut June 8th (https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-cadence) 🙃

But the CUDA 13 changes we were hoping for did make it in, so we could still update RAPIDS to building against CTK 13.3 (#268) and have the next RAPIDS release be compatible with the latest torch.

Some evidence of this (ref: https://dev-discuss.pytorch.org/t/pytorch-2-13-final-rc-available/3410):

docker run \
  --rm \
  -it python:3.13 \
  bash

PIP_ARGS=(
  --dry-run
  --prefer-binary
  --extra-index-url https://pypi.anaconda.org/rapidsai-wheels-nightly/simple/
)

CUDA12_ARGS=(
  --index-url https://download.pytorch.org/whl/test/cu126
  'cudf-cu12>=26.8.0a0'
  'cugraph-cu12>=26.8.0a0'
  'cugraph-pyg-cu12>=26.8.0a0'
  'cuml-cu12>=26.8.0a0'
  'cuvs-cu12>=26.8.0a0'
  'nvidia-nvjitlink-cu12>=12.9,<13.0'
)

CUDA13_ARGS=(
  --extra-index-url https://download.pytorch.org/whl/test/cu130
  --extra-index-url https://download.pytorch.org/whl/test/cu132
  'cudf-cu13>=26.8.0a0'
  'cugraph-cu13>=26.8.0a0'
  'cugraph-pyg-cu13>=26.8.0a0'
  'cuml-cu13>=26.8.0a0'
  'cuvs-cu13>=26.8.0a0'
  'nvidia-nvjitlink>=13.3,<14.0'
)

❌ CUDA 12.6 + PyTorch 2.13 + RAPIDS 26.08 + nvidia-nvjitlink-cu12>=12.9

$ pip install "${PIP_ARGS[@]}" "${CUDA12_ARGS[@]}" 'torch==2.13.0+cu126'
ERROR: Cannot install cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.6.3 and nvidia-nvjitlink-cu12<13.0 and >=12.9 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested nvidia-nvjitlink-cu12<13.0 and >=12.9
    cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx] 12.6.3 depends on nvidia-nvjitlink-cu12==12.6.85.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "nvjitlink"

✅ CUDA 13.0 + PyTorch 2.13 + RAPIDS 26.08 + nvidia-nvjitlink>=13.3

$ pip install "${PIP_ARGS[@]}" "${CUDA13_ARGS[@]}" 'torch==2.13.0+cu130'
Would install ... cuda-bindings-13.3.1 cuda-core-1.0.1 ... cuda-toolkit-13.0.3.0 cudf-cu13-26.8.0a852 cugraph-cu13-26.8.0a38 cugraph-pyg-cu13-26.8.0a21 cuml-cu13-26.8.0a124 cupy-cuda13x-14.1.1 cuvs-cu13-26.8.0a82 ... nvidia-nvjitlink-13.3.33 ... torch-2.13.0+cu130 ...

✅ CUDA 13.2 + PyTorch 2.13 + RAPIDS 26.08 + nvidia-nvjitlink>=13.3

$ pip install "${PIP_ARGS[@]}" "${CUDA13_ARGS[@]}" 'torch==2.13.0+cu132'
Would install ... cuda-bindings-13.3.1 cuda-core-1.0.1 ... cuda-toolkit-13.2.1 cudf-cu13-26.8.0a852 cugraph-cu13-26.8.0a38 cugraph-pyg-cu13-26.8.0a21 cuml-cu13-26.8.0a124 cupy-cuda13x-14.1.1 cuvs-cu13-26.8.0a82 ... nvidia-nvjitlink-13.3.33 ... torch-2.13.0+cu132 ...

[jameslamb]

I've put the target date for this in v26.10 (the next release) and added some tasks to the original post.

Once get a release with pytorch/pytorch#186958 included for torch's CUDA 12 wheels, all RAPIDS CUDA 12 and 13 wheels would be installable with the latest torch CUDA wheels 😁

I'll keep an eye on the torch issue tracker for a 2.13.1 release, and if one comes up I'll ask for that PR to be cherry-picked onto it.