Skip to content

Cover CUDA 13.4 in the Windows CUDA pull request jobs - #25307

Merged
Gasoonjia merged 11 commits into
mainfrom
gasoonjia/windows-cuda-13.4
Nov 21, 2026
Merged

Gasoonjia merged 11 commits into
mainfrom
gasoonjia/windows-cuda-13.4

Conversation

@Gasoonjia

@Gasoonjia Gasoonjia commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #25255. Rebased onto #25284 (e1390b6acb), so the 13.4 rows now install the torch 2.15.0 release, and the regenerated aoti_cuda_shims.lib has aoti_torch_dtype_element_size. Also triggers on torch_pin.py, since aoti_cuda_shims.lib has to move with the torch pin.

Why

PyTorch publishes CUDA 13.2 and 13.4 wheels (torch 2.15.0 is on whl/cu132 and whl/cu134, not on whl/cu130), and ExecuTorch's SUPPORTED_CUDA_VERSIONS is (13, 2), (13, 4). But every job in cuda-windows.yml ran 13.2 only, because the executorch-ubuntu-22.04-cuda-windows image carries one train's toolkits. The only 13.4 Windows cross-compile was in the nightly wheel pipeline, so a pull request that broke it could land green.

What

  • Model export and e2e: each row names its train (cuda-version, the same key as cuda.yml and build-wheels-cuda-windows.yml).
    • If the row's train differs from the image's nvcc, the export fetches that train's Linux nvcc and Windows cudart with install_cuda_redist.py, as the wheel pipeline does. The image is unchanged.
    • The export then asserts that install_executorch.sh picked a torch built for that train.
    • Artifact names carry the train.
  • 13.4 rows: each train installs its own toolkit and its own torch, so 13.4 runs two models; all six rows (four models) keep running on 13.2:
    • dinov2, the smallest model;
    • Voxtral Realtime quantized-int4-tile-packed, the only quant here that lowers to _weight_int4pack_mm and so runs the int4mm shim built by that train's nvcc. It already runs on 13.2.
  • e2e gate: e2e is not gated on the export matrix succeeding, mirroring build-wheels-cuda-windows.yml. One row's failed export, say one model's, no longer skips the other rows; each row fails on its own missing artifact. A skipped export (fork PR) still skips e2e.
  • unittest-cuda-runtime-windows: runs on both trains. The Windows GPU runners already carry the v13.4 toolkit.
  • PATH: both Windows jobs put CUDA_HOME\bin\x64;CUDA_HOME\bin first on PATH, before setup, as test_cuda_runtime_windows.ps1 and the wheel e2e script do.
  • Path filters: the workflow triggers on .ci/scripts/wheel/install_cuda_redist.py, and on torch_pin.py, install_requirements.py and install_utils.py. TORCH_VERSION lives in torch_pin.py, which install_requirements.py reads, and install_utils.py holds SUPPORTED_CUDA_VERSIONS. The torch pin is there because aoti_cuda_shims.lib has to move with it.
  • Check names: export and e2e jobs are named <job>-<cuda-version> (<model>, <quant>). GitHub cuts generated names at 100 characters, which gave the 13.2 and 13.4 Voxtral Realtime rows the same name.
  • test_cuda_workflow.py: reads each job's train from the matrix key its gpu-arch-version names, and checks that:
    • each Windows job covers every supported train and uses it (gpu-arch-version, the CUDA\v path, bin\x64 before setup);
    • export and e2e list the same rows;
    • every train has a tile-packed int4 row;
    • every model runs on the image's own train;
    • the export fetches both the Linux toolkit and the Windows cudart and exports WINDOWS_CUDA_HOME;
    • e2e doesn't gate on the export matrix's overall success.

Test plan

  • pytest .ci/scripts/tests/test_cuda_workflow.py .ci/scripts/tests/test_filter_cuda_matrix.py: 36 passed.
  • Reintroduced each guarded regression locally, and each one fails a test:
    • e2e gate back to == 'success';
    • runtime pinned to v13.2 or gpu-arch-version: "13.2";
    • no bin\x64 on PATH (e2e or runtime);
    • no Windows cudart fetch;
    • image bumped to 13.4;
    • the 13.4 int4 row switched back to weight-only;
    • an exclude that names other keys along with cuda-version, which could drop every row of a train;
    • an explicit name that loses the row values, or goes over 100 characters.
  • Full .ci/scripts/tests has the same 36 environment failures/errors (missing local deps or env vars) with and without this change.
  • install_cuda_redist.py --train 13.4 --platform windows-x86_64 --components cuda_cudart gives bin/x64/cudart64_13.dll and lib/x64/cudart.lib. Those are the paths torch.utils.cpp_extension (WINDOWS_CUDA_HOME/lib/x64) and Inductor's MinGW import-library step (bin/x64) read.
  • CI on c8d4223536 (rebased onto Use torch 2.15.0 for every CUDA train #25284, torch 2.15.0): all 364 checks pass.
    • Exports: both 13.4 exports pass, Voxtral Realtime tile-packed int4 and dinov2. The tile-packed export now links aoti_torch_dtype_element_size from the regenerated aoti_cuda_shims.lib.
    • Windows e2e: both 13.4 rows pass, Voxtral Realtime tile-packed int4 (which runs the int4mm shim built by 13.4's nvcc) and dinov2. All six 13.2 rows pass too.
    • Runtime: unittest-cuda-runtime-windows-13.2 and -13.4 pass. The 13.4 runtime job built with v13.4\bin\nvcc.exe.

@pytorch-bot

pytorch-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25307

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit e00a742 with merge base e1390b6 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 19, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 28m 19s —— View job


Request Changes: the new CUDA 13.4 export row shares the e2e job's matrix-wide result == 'success' gate, so if only the 13.4 export fails, all six 13.2 Windows e2e rows are skipped too. That can happen soon: the cu134 nightly torch pin from 2026-09-21 reaches download.pytorch.org's ~60-day retention about now. The other findings are smaller gaps in path filters, Windows PATH setup, the 13.4 scope comment, and what the new tests check.

Detailed review

PR Review: #25307

Summary

The PR adds a CUDA 13.4 dinov2 row to the Windows export and e2e matrices, runs unittest-cuda-runtime-windows on both trains, and extends test_cuda_workflow.py to check train coverage. When the image's nvcc doesn't match, the new row fetches the 13.4 Linux nvcc and Windows cudart with install_cuda_redist.py. Problems found:

  • The new row can skip every e2e row.
  • The script it now depends on is outside the workflow's path filters.
  • The 13.4 e2e row may load the 13.2 CUDA runtime (low confidence).
  • The "only the toolkit differs" scope claim isn't accurate.
  • The new tests check which trains the matrices list, not that the jobs use them.

Infrastructure

1. The new 13.4 export row can skip all seven e2e rows (.github/workflows/cuda-windows.yml:199)

needs.export-model-cuda-windows-artifact.result is one result for the whole export matrix. If any export leg fails, test-model-cuda-windows-e2e skips every row, including the 13.2 rows whose artifacts uploaded fine. The gate isn't new, but before this PR every row shared one train and one torch. The 13.4 row adds failure modes the other rows don't have:

  • Toolkit downloads: lines 152-159 download two toolkits, and _fetch in .ci/scripts/wheel/install_cuda_redist.py:55-59 has no retry.
  • Expiring torch pin: the row installs torch==2.15.0.dev20260921+cu134. install_requirements.py:30-34 says this pin expires because download.pytorch.org keeps nightlies for about 60 days, which runs out around 2026-11-20. From then on, every run loses all Windows e2e coverage until someone bumps the pin.

build-wheels-cuda-windows.yml:253-258 stopped gating on the matrix result for this exact reason. Suggested gate: !cancelled() && needs.export-model-cuda-windows-artifact.result != 'skipped' && (...). The skipped check still covers the fork-PR case described in the comment at lines 192-196, and only rows whose own export failed fail at download. Update that comment too. Fix this →

2. .ci/scripts/wheel/install_cuda_redist.py isn't in this workflow's path filters

The export job now runs this script for the 13.4 row, but the script is missing from pull_request.paths (lines 15-26) and from the contains() lists at lines 59-69 and 198-208. A PR that changes only this script, for example _RELEASES["13.4"] at install_cuda_redist.py:36, gets no 13.4 coverage:

  • build-wheels-cuda-windows.yml does trigger on .ci/**, but its PR matrix is cu132 only (.github/scripts/filter_cuda_matrix.py:76).
  • On push, this job runs only on sampled full runs.

Fix this →

3. (Low confidence) The 13.4 e2e row may load the 13.2 CUDA runtime

Line 269 prepends only $env:CUDA_HOME\bin. The 13.x Windows toolkits keep their DLLs in bin\x64, which is why test_cuda_runtime_windows.ps1:27 and wheel/test_cuda_windows_e2e.ps1:59 prepend bin\x64;bin. test_model_e2e_windows.ps1 doesn't add it either.

Every 13.x train names its runtime cudart64_13.dll. So which copy the runner and the AOTI DLL load depends on the order of the image's system PATH, unless dinov2's post-build copy (examples/models/dinov2/CMakeLists.txt:88-99) picks up cudart. I couldn't confirm whether TARGET_RUNTIME_DLLS includes it.

If the 13.2 copy wins, the row still passes, because -ExpectedCudaVersion (line 278) checks only nvcc. Prepending bin\x64 the way the sibling scripts do settles it. Fix this →

Testing

4. The 13.4 row covers less than its comment says (lines 109-110 and 244-245)

The comment and the PR description say only the toolkit differs between trains. The Linux export also installs a different torch per train (install_requirements.py:33-37, 104-130; test_cu134_dependencies.py:72, 114):

  • 13.2: torch==2.14.0
  • 13.4: torch==2.15.0.dev20260921+cu134

The model also decides which backend kernels run. So on 13.4, neither the int4 export path under that torch nor backends/cuda/runtime/shims/int4mm.cu built by the 13.4 nvcc ever runs, and unittest-cuda-runtime-windows doesn't cover int4 either.

Either add one quantized 13.4 row, for example nvidia/parakeet-tdt with quantized-int4-weight-only, or narrow the comment. "The other CUDA train" also assumes there are exactly two trains. Fix this →

5. The new tests check which trains the matrices list, not that the jobs use them

test_windows_jobs_cover_every_supported_train compares only matrix values. If line 305 goes back to "13.2" or line 314 goes back to v13.2, every test still passes, and unittest-cuda-runtime-windows-13.4 quietly runs 13.2. Two changes:

  • Check the wiring: for the three Windows jobs, mirror line 55's gpu-arch-version == "${{ matrix.cuda }}", and assert that the e2e and runtime scripts use CUDA\v${{ matrix.cuda }}.
  • Stop guessing the key: _matrix_cuda_versions (lines 28-32) should read the key named in the gpu-arch-version expression. Today any include wins, which breaks in two ways:
    • Moving test-model-cuda-e2e (cuda.yml:1043) onto a cuda-version axis, as test-cuda-builds uses, raises KeyError on its pybind include rows.
    • A fromJSON(...) matrix, the pattern at build-wheels-cuda-windows.yml:170, gets split into single characters.

Fix this →

6. test_windows_export_image_matches_the_export_job no longer checks what its name says

It used to assert gpu-arch-version == image_version. Line 114 now asserts only that the image's version appears somewhere in the matrix, which line 109 plus the coverage test already guarantee. Two regressions now pass:

  • Bumping .ci/docker/build.sh:91 to CUDA_VERSION=13.4 passes, while the six 13.2 rows quietly start downloading two toolkits on every run.
  • Deleting the Windows cudart fetch (lines 158-160) passes the substring checks at lines 116-117, because the --train substring also matches the Linux install at line 152. The 13.4 export would then link against the image's 13.2 Windows cudart.

Assert that the non-dinov2 rows use image_version, and check for --platform windows-x86_64 and the WINDOWS_CUDA_HOME export. Fix this →

Specific Comments

  • .github/workflows/cuda-windows.yml:82 (every row): the matrix key here is cuda, but cuda.yml:155 and build-wheels-cuda-windows.yml:170 use cuda-version. Using one name would also let _matrix_cuda_versions drop a fallback.
  • .ci/scripts/tests/test_cuda_workflow.py:73: for version in versions rebinds version from line 66.
  • .ci/scripts/tests/test_cuda_workflow.py:93-99: the generator rebinds name from the loop at line 83. Two direct lookups would be simpler.
  • .ci/scripts/wheel/install_cuda_redist.py:10, 33-35: these lines say the script's only consumers are the wheel e2e jobs, which take their matrix from filter_cuda_matrix.py. cuda-windows.yml now uses it too, with a hard-coded matrix.

Recommendation

Request Changes. Finding 1 is a small if: change, and the expiring cu134 pin makes it likely to trigger soon. Finding 2 adds one path in three places. The rest are up to the author.

Review steps
  • Gather context (diff, related workflows, scripts)
  • Run three independent reviewers
  • Consolidate findings
  • Fact-check findings
  • Post review

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Addressed in a2e42ab:

  1. e2e gate: now !cancelled() && needs.export-model-cuda-windows-artifact.result != 'skipped', as in build-wheels-cuda-windows.yml. A failed 13.4 export no longer skips the 13.2 rows; each row fails on its own missing artifact. A fork PR, where the export is skipped, still skips this job.
  2. Path filters: .ci/scripts/wheel/install_cuda_redist.py added to pull_request.paths and to both contains() lists.
  3. PATH: the e2e job puts CUDA_HOME\bin\x64 ahead of bin, as test_cuda_runtime_windows.ps1 and test_cuda_windows_e2e.ps1 do.
  4. 13.4 scope: added nvidia/parakeet-tdt quantized-int4-weight-only on 13.4, since each train installs its own torch and only a quantized model runs the int4 kernels. Also rewrote the comment so it doesn't assume exactly two trains.
  5. Tests check the wiring:
    • _matrix_cuda_versions reads the key named in gpu-arch-version.
    • The Windows jobs must use ${{ matrix.cuda-version }} for gpu-arch-version and the CUDA\v path.
    • The e2e PATH must include bin\x64.
    • Every model must run on the image's train.
    • The export must fetch the Windows cudart and export WINDOWS_CUDA_HOME.
  6. New checks fail on the regressions they guard: I reintroduced each one locally (gate back to == 'success', runtime pinned to v13.2, gpu-arch-version: "13.2", no bin\x64, no Windows cudart fetch, image bumped to 13.4), and every one makes a test fail.

Also:

  • The matrix key is now cuda-version, matching cuda.yml and the wheel workflow.
  • The version/name rebinding in the tests is gone.
  • install_cuda_redist.py's comments now name cuda-windows.yml as a consumer.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 32m 19s —— View job


Request Changes: the CUDA 13.4 int4 row added in a2e42ab (parakeet-tdt quantized-int4-weight-only, which my previous review wrongly suggested) never calls _weight_int4pack_mm, so no int4 kernel runs on 13.4 even though the new matrix comment says one does; the other findings are minor.

Detailed review

PR Review: #25307

Summary

The PR runs the Windows CUDA export, e2e and runtime-test jobs on both 13.2 and 13.4. The export job fetches the 13.4 toolkits itself when the image's nvcc doesn't match, and test_cuda_workflow.py now checks that wiring. Problems found in a2e42ab:

  • The new 13.4 int4 row doesn't run the int4 kernel it was added to cover.
  • The runtime job may install torch for the wrong train (low confidence).
  • One assertion is duplicated, one comment is worded wrongly, and the PR description doesn't match the diff.

Infrastructure

The runtime job installs torch before it puts the row's toolkit on PATH (.github/workflows/cuda-windows.yml:328-330)

The two jobs set up PATH differently:

  • e2e: puts CUDA_HOME\bin\x64;CUDA_HOME\bin first on PATH before running setup-windows.ps1 (:283).
  • runtime: sets only CUDA_HOME and CUDA_PATH. .ci/scripts/test_cuda_runtime_windows.ps1:27 fixes PATH, but only after setup has run.

_selected_nvcc checks PATH before CUDA_PATH (install_utils.py:213-220). So on a row whose train isn't the runner's default nvcc, install_executorch.bat installs the other train's torch. The C++ tests are then built with the row's nvcc (test_cuda_runtime_windows.ps1:24-27, 45-46), while find_package_torch() (backends/cuda/CMakeLists.txt:214) finds the other train's torch.

Low confidence: I couldn't read test-infra's windows_job.yml to check whether gpu-arch-version already reorders PATH. Copying line 283 into this job settles it either way.

Related: the e2e PATH assertion at .ci/scripts/tests/test_cuda_workflow.py:111-115 is repeated word for word at :116-120. Pointing the second copy at this job would test the fix. Fix this →

Testing

The 13.4 int4 row never calls _weight_int4pack_mm (.github/workflows/cuda-windows.yml:111-121, :253-263)

The new comment says this row "runs the int4 kernels the non-quantized model does not". It doesn't:

  • quantized-int4-weight-only passes --qlinear_encoder 4w with no packing format (.ci/scripts/export_model_artifact.sh:303).
  • extension/llm/export/quantize.py:50-54 turns that into IntxWeightOnlyConfig(weight_dtype=torch.int4), not Int4WeightOnlyConfig. The encoder's linears become IntxUnpackedToInt8Tensor, which lowers to a dequantize followed by a plain matmul.
  • The Triton int4 path (pack_int4_weights_for_aoti) is used only by Voxtral Realtime, and only on ROCm (examples/models/voxtral_realtime/export_voxtral_rt.py:758).

So no 13.4 job runs backends/cuda/runtime/shims/int4mm.cu (compiled by the 13.4 nvcc) or any Triton int4 kernel. .ci/scripts/test_cuda_runtime_windows.ps1:32-42 has no int4 test either. My previous review gave this exact row as its example, and that example was wrong.

quantized-int4-tile-packed is the quant that selects Int4WeightOnlyConfig with tile_packed_to_4d (export_model_artifact.sh:296), which lowers to _weight_int4pack_mm. Options:

  • nvidia/parakeet-tdt quantized-int4-tile-packed on 13.4. It runs on Linux (.github/workflows/cuda.yml:991-994 excludes only parakeet's weight-only row) but has no Windows row yet. .ci/scripts/tests/test_cuda_workflow.py:147-154 then needs a 13.2 row for it too.
  • mistralai/Voxtral-Mini-4B-Realtime-2602 quantized-int4-tile-packed vr-offline on 13.4. It already runs on Windows with 13.2 and passes that test as it is, but it is a much larger export.
  • Keep the row and reword both comments. Say it covers an int4 weight-only export under the 13.4 torch, not the int4 kernels.

Fix this →

Specific Comments

  • .github/workflows/cuda-windows.yml:199: the comment says "Not gated on the export's result", but line 207 does check that result (!= 'skipped'). "Not gated on the export succeeding" says what is meant.
  • PR description: it becomes the squash commit message but still describes 494823e:
    • The matrix key is cuda-version, not cuda.
    • The "13.4 scope" bullet says 13.4 runs on dinov2 only, and still says only the toolkit differs between trains. The workflow comment no longer makes that claim.
    • It says the jobs "together" cover every train; the test now requires each job to cover every train.
    • "53 passed" is from before a2e42ab changed the tests.

Recommendation

Request Changes. The Testing finding leaves open the coverage gap that the row was added to close, and the comment says the gap is closed. Swapping the row or rewording the comment resolves it. The Infrastructure finding is low confidence, and the rest is optional.

Review steps
  • Gather context (diff, a2e42ab, related scripts)
  • Run three independent reviewers
  • Consolidate findings
  • Fact-check findings
  • Post review

Not run: python and pytest, which this action's --allowedTools blocks, and fetching test-infra's windows_job.yml, which needs WebFetch. This review comes from reading the code. This PR's CI was still running when it was posted.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Addressed in 1c3cfb3:

  • The 13.4 int4 row didn't run an int4 kernel: confirmed. quantized-int4-weight-only passes no packing format, so quantize.py picks IntxWeightOnlyConfig, which lowers to dequantize plus matmul. The 13.4 int4 row is now Voxtral Realtime quantized-int4-tile-packed (vr-offline). That is the quant that selects Int4WeightOnlyConfig(tile_packed_to_4d) and lowers to _weight_int4pack_mm, so it runs the int4mm.cu shim built by the 13.4 nvcc. It already runs on Windows on 13.2 (its export takes ~29 min), so the 13.4 row differs only in its train. I picked it over parakeet tile-packed, which has no Windows row yet. A test now requires a tile-packed row on every train.
  • Runtime job's PATH: the runtime job now puts CUDA_HOME\bin\x64;CUDA_HOME\bin first on PATH before setup-windows.ps1, like e2e. The test checks this for both jobs and that the line comes before setup. In practice this job installs CPU torch (torch==2.14.0 from whl/test/cpu on the previous head), and its 13.4 run already built with v13.4\bin\nvcc.exe. So nothing was mis-built, but the two jobs now agree.
  • Duplicated assertion: removed.
  • Gate comment: now says "not gated on the export succeeding".
  • PR description: rewritten to match the current diff (cuda-version key, the two 13.4 rows, per-job coverage, 54 tests).

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 28m 50s —— View job


I found nothing blocking: 1c3cfb3 resolves every round 2 finding and only two wrong comments and an order-dependent slice in the test remain, so I recommend Approve once the 13.4 rows pass (they were still running when I reviewed).

Detailed review

PR Review: #25307

Summary

This PR runs the Windows CUDA pull request jobs (export, e2e and runtime unit tests) on 13.4 as well as 13.2. It adds a cuda-version to each matrix row, fetches a redist toolkit for any train the export image doesn't carry, gates each e2e row on its own artifact, and adds tests that tie the matrices to SUPPORTED_CUDA_VERSIONS. Nothing blocks: the round 2 findings are resolved, and only the three notes below remain.

Specific Comments

  • .ci/scripts/tests/test_cuda_workflow.py:111-112: the comment says the toolkit leads PATH "before setup picks a torch for its nvcc". That isn't what happens. On Windows, determine_torch_url always returns CPU torch (install_utils.py:368-372), as your 1c3cfb3 reply says. What PATH order decides is which nvcc setup.py finds and builds the CUDA _C with (setup-windows.ps1:21-26), because PATH is checked before CUDA_PATH (install_utils.py:213-220). Fix this →
  • .ci/scripts/tests/test_cuda_workflow.py:106: WINDOWS_TRAIN_JOBS[1:] only means "the jobs that run on a Windows runner" while the export job stays first in the tuple. An explicit tuple of test-model-cuda-windows-e2e and unittest-cuda-runtime-windows would say that directly. Fix this →
  • .github/workflows/cuda-windows.yml:339-340: "about 18 GB per run" was written before this PR added a second Voxtral-Mini-4B-Realtime int4 export and a 13.4 dinov2 export, so the figure is now too low. Fix this →

Recommendation

Approve. Nothing blocks; the notes above are about wording.

The 13.4 rows haven't been shown to work yet. When I reviewed, every export and runtime job for 1c3cfb3 was still running and no e2e row had started. That leaves two things unconfirmed: the 13.4 Windows toolkit (CUDA\v13.4 on windows.g5.4xlarge.nvidia.gpu) and the 13.4 tile-packed Voxtral Realtime run.

Heads-up: CU134_TORCH_PACKAGES pins the 2026-09-21 torch nightly. install_requirements.py:31 says nightlies are kept for about 60 days, so around 2026-11-20 both 13.4 export rows will start failing at install, as will every other cu134 job, until the pin is moved.

Review steps
  • Gather context (diff vs 5e70618, round 1 and 2 threads, the author's 1c3cfb3 reply)
  • Run three independent reviewers over every changed line
  • Consolidate findings
  • Fact-check findings
  • Write up the review

Not run:

  • pytest .ci/scripts/tests/test_cuda_workflow.py: this runner needs approval for pytest and python -c.
  • test-infra's linux_job_v3.yml and windows_job.yml couldn't be fetched, so I couldn't check what gpu-arch-version: 13.4 sets up on each runner.

@Gasoonjia
Gasoonjia force-pushed the gasoonjia/windows-cuda-13.4 branch from a874e5e to c8d4223 Compare November 20, 2026 23:06
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Rebased onto main after #25284 landed (e1390b6acb). The 13.4 rows now get the torch 2.15.0 release and the regenerated shims .lib. Also added: torch_pin.py in the paths and the export/e2e/runtime triggers; the _matrix_cuda_versions exclude/include fix (only single-key excludes, include after exclude, axis-is-a-list assert); and the wording points from the last review. 52 workflow tests pass.

@claude review effort=max

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 27m 36s —— View job


I'd approve once the 13.4 rows in the running CI pass on torch 2.15.0. Nothing blocks: what's left is a minor gap in _matrix_cuda_versions, which is code my last review suggested, plus wording written before the rebase, mostly in the PR description.

Detailed review

PR Review: #25307

Summary

The PR runs the Windows CUDA export, e2e and runtime jobs on CUDA 13.4 as well as 13.2. The 13.4 export rows fetch their own Linux nvcc and Windows cudart with install_cuda_redist.py. test_cuda_workflow.py checks that every job covers every supported train.

Nothing blocks. One minor test-helper gap and some stale wording remain. The 13.4 Voxtral Realtime export and both 13.4 e2e rows haven't passed in CI yet. Run 44834278062 for c8d4223 is still running.

Testing

  • _matrix_cuda_versions can count a train as covered when none of its rows run (.ci/scripts/tests/test_cuda_workflow.py:38-41, minor).

    • What it does: the helper ignores exclude rows that name cuda-version along with other keys. The comment says that "can only add a version that never runs, never skip one that does".
    • Why that matters: adding a version that never runs is harmless for test_every_cuda_job_uses_a_supported_version. For test_windows_jobs_cover_every_supported_train, it is exactly the failure the test exists to catch.
    • Example: give unittest-cuda-runtime-windows a second axis config: [release] and exclude: [{cuda-version: "13.4", config: release}]. No 13.4 job runs, but the helper still returns both trains, so the coverage test passes.
    • Where it came from: my last review proposed this code and called it harmless. That was wrong for the coverage test.
    • Impact today: no job has an exclude like this, so it only matters for a future edit.

    Fail on such rows instead of skipping them:

        # GitHub drops a combination only when it matches every key of an exclude row, and
        # applies include after exclude.
        excludes = [row for row in matrix.get("exclude", []) if key in row]
        assert all(row.keys() == {key} for row in excludes), excludes
        excluded = {row[key] for row in excludes}

    Fix this →

Specific Comments

  • .github/workflows/cuda-windows.yml:207-208: the example "(say a train whose torch pin expired)" describes the per-train cu134 nightly pin that Use torch 2.15.0 for every CUDA train #25284 removed. Both trains now install torch_pin.py's 2.15.0. A failure confined to one row would fit better, say one model's export failing. The description's e2e gate bullet has the same example ("say an expired cu134 torch pin"). Fix this →
  • The PR description was written before the rebase:
    • The "Land after Use torch 2.15.0 for every CUDA train #25284." paragraph and its CU134_TORCH_PACKAGES nightly no longer apply.
    • The Path filters bullet leaves out torch_pin.py. It also says install_requirements.py and install_utils.py "hold the torch pins". TORCH_VERSION now lives in torch_pin.py, which install_requirements.py:23-25 reads.
    • The test plan runs .ci/scripts/tests/test_cu134_dependencies.py, which no longer exists. It also reports 54 passed, while your comment says 52.
    • The CI evidence comes from before the rebase. It covers the 13.4 dinov2 export and both runtime jobs. It does not cover the 13.4 Voxtral Realtime tile-packed export or either 13.4 e2e row.

Recommendation

Approve once the 13.4 rows in run 44834278062 pass. No 13.4 row has passed yet on torch 2.15.0+cu134 with the regenerated shims .lib. The 13.4 Voxtral Realtime tile-packed export and both 13.4 e2e rows have no passing CI result at all. The _matrix_cuda_versions assert is minor and can land in this PR or later. Please refresh the description before merging.

Review steps

  • Gather context: the diff against origin/main, the rebase onto Use torch 2.15.0 for every CUDA train #25284 (e1390b6acb), and the earlier review rounds
  • Run three independent reviewers over all three changed files
  • Consolidate and fact-check their findings. All three reviewers worried that the image's LD_LIBRARY_PATH still puts the 13.2 /usr/local/cuda/lib64 first for the 13.4 export. I dropped that concern. PyTorch's CUDA wheels record their nvidia library directories as DT_RPATH, and the loader searches that before LD_LIBRARY_PATH. I couldn't inspect the 2.15.0 wheel here, so the running 13.4 export is the real check.
  • Check CI: run 44834278062 is still running, and no 13.4 row has finished.
  • Run pytest .ci/scripts/tests/test_cuda_workflow.py: not possible in this environment, so this review is from reading the code only.

_matrix_cuda_versions asserted nothing about exclude rows that name other keys
along with the CUDA version; one could drop every row of a train and the
coverage test would still count it. Reject them instead. The e2e gate comment's
example was the per-train nightly pin #25284 removed.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Addressed in f7aef94:

  • _matrix_cuda_versions: it now asserts that every exclude row naming cuda-version names nothing else, instead of skipping such rows. Your example (config: [release], exclude: [{cuda-version: "13.4", config: release}]) now fails the helper, and a single-key exclude still drops its train.
  • e2e gate comment: the example is now one model's export failing, not the cu134 pin Use torch 2.15.0 for every CUDA train #25284 removed.
  • Description: refreshed. It drops test_cu134_dependencies.py, adds torch_pin.py to the path filters, and gives the current counts and CI.

CI on c8d4223: all 364 checks pass.

  • Both 13.4 exports pass, including the Voxtral Realtime tile-packed export.
  • Both 13.4 e2e rows pass, plus all six 13.2 rows.
  • unittest-cuda-runtime-windows-13.2 and -13.4 pass.

This push changes only the test helper and a comment.

@claude review effort=max

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 28m 42s —— View job


Review of f7aef94: I recommend Approve once run 44854278127 finishes green; nothing blocks, and the details below have one minor test gap, one question about the path filters and four wording nits.

Detailed review

PR Review: #25307

Summary

The PR gives the Windows CUDA export, e2e and runtime jobs a cuda-version axis. 13.4 gets the dinov2 row, the Voxtral Realtime tile-packed row and its own runtime job, and its exports fetch the 13.4 redist toolkits onto the 13.2 image.

There are no blocking issues.

f7aef94 changes only the test helper and a comment (git diff --stat c8d4223 f7aef94), and every job in c8d4223's run passed. Run 44854278127 for f7aef94 is still going:

  • 7 of 8 exports have passed, including both 13.4 exports.
  • The last 13.2 export (Voxtral-Mini-3B int4-weight-only), the e2e rows and both runtime jobs haven't finished.

Infrastructure

  • Question (the gap predates this PR, non-blocking): the filters now list install_utils.py but not setup.py. setup.py loads it (setup.py:72-79) and builds the CUDA backend from it:

    • the train check against SUPPORTED_CUDA_VERSIONS (:778-843);
    • the toolkit root from _selected_nvcc() (:908-911);
    • the Windows nvcc.exe (:2723).

    A PR that touches only that code runs none of these jobs. The Windows wheel workflow builds one train on a pull request (build-wheels-cuda-windows.yml:138-141), so the other train isn't built on Windows until a sampled full run or the nightly wheel build. If setup.py is left out on purpose because it changes often, a comment beside the filter list would say so.

Testing

  • .ci/scripts/tests/test_cuda_workflow.py:193-207: the test pins the toolkit fetch and export WINDOWS_CUDA_HOME, but not the torch-train assert at cuda-windows.yml:189.
    • That assert is the only line that checks the toolkit switch reached torch.
    • The lines it depends on are unpinned too: export PATH and export CUDACXX at :170-171. _selected_nvcc (install_utils.py:174) reads them to choose the nvcc, and that nvcc decides the torch URL.
    • If the assert is dropped, every test still passes. Nothing would then catch a 13.4 row exported with the image's cu132 torch: Windows installs CPU torch (install_utils.py:368-372), so the e2e never sees which torch did the export.
    • test_cuda_probe_checks_the_result_and_torch_train already covers the equivalent probe in test-cuda-build.sh.
    • Fix: self.assertIn("assert v == '${{ matrix.cuda-version }}'", script). Fix this →

Specific Comments

  • .github/workflows/cuda-windows.yml:117-118: "The other trains" has nothing to refer to after "Each train…". Something like "Trains other than the image's (13.2) get only…" fixes it. This was my wording from last round.
  • .github/workflows/cuda-windows.yml:159-160: "and the Windows runtime it links" reads as a second thing nvcc decides. build-wheels-cuda-windows.yml:197-198 says "and its own Windows runtime to link against".
  • .github/workflows/cuda-windows.yml:208-209: line 208 breaks early. "The skipped check" reads like a skipped CI check; "The result != 'skipped' condition keeps…" says which line is meant.
  • .ci/scripts/tests/test_cuda_workflow.py:172: test_windows_export_image_matches_the_export_job is named after the single gpu-arch-version comparison this PR removed. It now checks that every model has a row on the image's train and that the other trains fetch their own toolkits; one option is test_windows_export_fetches_toolkits_the_image_lacks.

Fix these comments →

Recommendation

Approve. Nothing blocks. The test gap is a one-line assertIn, and the setup.py question is a policy call. This assumes run 44854278127 finishes green.

Review steps

  • Diffed f7aef94 against the PR base e1390b6 with git diff e1390b6 HEAD. The shallow checkout has no merge base with origin/main, and git fetch needed approval.
  • Three independent reviewers covered all three changed files. I merged their findings and re-read every cited line.
  • Dropped on fact-check:
    • The 'torch_pin.py' substring also matches .github/scripts/update_pytorch_pin.py. pull.yml, trunk.yml, windows-msvc.yml and vulkan.yml match it the same way, so fixing it isn't this PR's job.
    • The new exclude assert in _matrix_cuda_versions can't trip on cuda.yml. The only job it reads there, test-cuda-builds, has no exclude.
    • Nits on lines this PR doesn't change (:182, :349).
  • Not done:
    • test_cuda_workflow.py was not run here, because running it needed approval.
    • test-infra's linux_job_v3.yml and windows_job.yml aren't in the checkout, so I took their behavior from the workflow comments.
    • Run 44854278127 was unfinished.

…ording

Only the export's torch-train assert sees which torch did a 13.4 export, since
the Windows e2e runs CPU torch, so the test now pins it along with the PATH and
CUDACXX exports that choose the nvcc. The export test is renamed for what it
checks now, and three comments are reworded.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Addressed in the latest push:

  • Torch-train assert: test_windows_export_fetches_toolkits_the_image_lacks now pins assert v == '${{ matrix.cuda-version }}', plus the export PATH and export CUDACXX lines that choose the nvcc. I removed the assert from the workflow locally to check: the test fails without it and passes with it.
  • Test name: renamed from test_windows_export_image_matches_the_export_job.
  • Wording: all three comments are fixed as you suggested:
    • "Trains other than the image's (13.2)";
    • "and its own Windows runtime to link against";
    • the e2e gate comment is reflowed and names the result != 'skipped' condition.
  • setup.py in the path filters: I've left it out. It's a policy call, and the gap predates this PR. I've asked the PR owner about it.

pytest .ci/scripts/tests/test_cuda_workflow.py .ci/scripts/tests/test_filter_cuda_matrix.py: 36 passed.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 28m 52s —— View job


Nothing blocks; there's one optional follow-up (the new toolkit switch copies the wheel pipeline's) and one comment-wording nit, so I recommend Approve. CI for e00a742 is still running, with nothing failed so far.

Detailed review

PR Review: #25307

Summary

The PR runs the Windows CUDA export, e2e and runtime jobs on both 13.2 and 13.4. Rows for a train other than the image's fetch that train's toolkit with install_cuda_redist.py. Nothing blocks; there are two minor points, below.

Code Quality

  • The toolkit switch is now in two workflows: .github/workflows/cuda-windows.yml:165-175 and .github/workflows/build-wheels-cuda-windows.yml:201-211, plus the torch-train assert at cuda-windows.yml:190 and build-wheels-cuda-windows.yml:217.
    • The two copies run the same commands. The only differences are this copy's nvcc check and the wheel pipeline's ${TRAIN} spelling.
    • An earlier revision of this PR already let them drift apart (LD_LIBRARY_PATH).
    • test_windows_export_fetches_toolkits_the_image_lacks pins only this copy. Any later change, such as a component a new train needs or another variable to export, has to be made twice by hand.
    • Suggested fix:
      • Move the block into one sourced script under .ci/scripts/wheel/. It takes the train and exports CUDA_HOME, PATH, CUDACXX and WINDOWS_CUDA_HOME.
      • cuda-windows.yml keeps its nvcc check around the source line.
      • The test pins the script, plus each workflow's source line.
    • A follow-up is fine; this doesn't block. Fix this →

Specific Comments

  • .ci/scripts/wheel/install_cuda_redist.py:34-37:
    • The current wording is "one entry per train in A, which the wheel end-to-end jobs take their matrix from, and in B". It reads as if "and in B" continues the relative clause.
    • Suggested wording: "one entry per train listed in filter_cuda_matrix.py's SUPPORTED_CUDA_VERSIONS (the wheel end-to-end jobs' matrix) or in cuda-windows.yml's matrices".
    • Fix this →

Recommendation

Approve. Nothing blocks, and both points above are optional. CI for e00a742 is still running. The workflow has had only comment changes since the all-green c8d4223.

Review steps

  • Gather context: the diff against e1390b6, the earlier rounds and the author's replies.
  • Run three independent reviewers. None of them reported a blocker. All three changed files were covered.
  • Consolidate findings.
  • Fact-check findings. I dropped the ones the code disproves.
  • Run test_cuda_workflow.py locally. Skipped: pytest needs approval in this sandbox, so I read the tests but did not run them.
  • Check CI for e00a742: 19 runs have passed, 12 are still running, and none have failed.

This branch was previously deployed

1 inactive deployment
cadence — e00a7420 Deployed Nov 21, 2026 by Gasoonjia via hifi-op-test / hifi4 #39871
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants