Repository navigation
Commit f8907c1
authored
Fix main ROCm CI: ProbeTest skip order, flaky SDPA pick test, drop torchcodec from voxtral (#25393)
Fix the two ROCm gfx950 jobs that fail on every `main` run.
### `unittest-rocm-gfx950`: `ModuleNotFoundError: No module named
'pandas'`
Every failing `main` run from `6b82a162e7` to `e7f6aefd09` has exactly
one failure:
`test_measured_routing.py::ProbeTest::test_real_measurements`. The last
green run was `fe24cf5777`.
**Why it errors instead of skipping.**
- `ProbeTest.setUp` (added by #25277) imports `executorch.export.utils`
before it calls `super().setUp()`.
- That import pulls in `executorch.devtools` and then
`devtools/inspector/_inspector.py`, which runs `import pandas`.
- `test-rocm-aoti.sh` installs the minimal wheel
(`EXECUTORCH_BUILD_MINIMAL=1`), which doesn't include pandas.
- The base class `_PassTest.setUp` already skips without Triton on CUDA.
That's why the other 162 tests in the file skip on ROCm, but `ProbeTest`
imports before it reaches that skip.
**Fix.** Call `super().setUp()` first, so `ProbeTest` skips on ROCm like
the rest of the file.
### `unittest-rocm-gfx950`: flaky
`SdpaCellStepAutotuneTest::test_compiled_pick_is_fastest_at_live_extents`
The test asserts that the config the autotuner picked at compile time is
within 1.10× of the fastest config, timed with CUDA-graph replays. It
failed in 2 of the last 13 `main` runs (`c346379f06`, `cd12c4d02d`),
both the same way:
- At live extent 4650, the pick was `BLOCK_M=16, BLOCK_N=64,
PACK_GQA=False` at about 119 µs.
- The best was the same tile with `PACK_GQA=True` at about 71.7 µs, a
1.66× regret.
**Cause: not confirmed.** It may be a real ROCm mis-pick: the
compile-time scenarios run on ROCm too, so a default ROCm export of this
SDPA can compile in the slower config. #25397 tracks two hypotheses: no
CUDA-graph autotune timing on ROCm, and the compile-time scenarios
timing an all-visible mask where this test times a sparse cell mask.
**Fix.** Deselect this one test on ROCm, next to the
`test_cuda_graph_timing.py` exclusion. The comment cites #25397, which
tracks re-enabling it.
### `test-voxtral-realtime-rocm-gfx950`: `Could not load libtorchcodec`
- `test-rocm-voxtral.sh` runs `conda install -c conda-forge ffmpeg`
without a version.
- Since Anaconda `main` published `ffmpeg-9.0.2` (2026-11-20 09:42Z),
that resolves to FFmpeg 9.
- The pinned `torchcodec==0.11.0` only loads FFmpeg 4–8. The log shows
it trying `libavutil.so.56`–`.so.60` and finding none.
- Before 09:42Z, the same unpinned install hung in conda's classic
solver ("Solving environment"). Runs were cancelled at the 180-minute
timeout every time, including #25284's.
- No commit in this repo caused it: neither the script nor `rocm.yml`
has changed since Oct 30.
**Fix: drop torchcodec and FFmpeg here.**
- Nothing in this job needs them. `run_rocm_e2e.sh` reads the clip with
`wave`, and nothing under `examples/models/voxtral_realtime` imports
torchcodec, `datasets` or `librosa`.
- `librosa` goes too: nothing in the job imports it.
- Their only user was the dataset fetch, and that dataset's rows are WAV
bytes. The fetch now reads them with `Audio(decode=False)` and decodes
with `soundfile`.
- This fixes both the load error and the solver hang. Pinning `ffmpeg<9`
would leave the hang.
### Test plan
- **Audio fetch:** the new snippet ran locally in a fresh venv with
`datasets==5.2.0` (CI's version), `soundfile`, and no torchcodec. It
wrote a 16 kHz, mono, 16-bit, 30.0 s WAV, the format `run_rocm_e2e.sh`'s
`wave` check expects.
- **`ProbeTest.setUp`:** confirmed with `ast` that its first statement
is now `super().setUp()`, and that `_PassTest.setUp` skips on
`torch.version.hip`.
- **Lint:** `black` 24.4.2 and `flake8` are clean, and `bash -n` passes
on both scripts.
- **CI on this PR:** I'll confirm both ROCm gfx950 jobs pass. I can't
run them here, since that needs an MI350.
**CI on 6549cb2:** `unittest-rocm-gfx950-rocm7.2` 719 passed, 163
skipped, 1 deselected, 0 failed, with
`ProbeTest::test_real_measurements` SKIPPED.
`test-voxtral-realtime-rocm-gfx950-rocm7.2` passed on MI350.1 parent e01c6a0 commit f8907c1
3 files changed
Lines changed: 15 additions & 10 deletions
File tree
- .ci/scripts
- backends/cuda/tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
149 | 149 | | |
150 | 150 | | |
151 | 151 | | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
152 | 155 | | |
153 | 156 | | |
154 | 157 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
68 | 68 | | |
69 | 69 | | |
70 | 70 | | |
71 | | - | |
| 71 | + | |
72 | 72 | | |
73 | 73 | | |
74 | | - | |
75 | | - | |
76 | | - | |
| 74 | + | |
77 | 75 | | |
78 | 76 | | |
79 | 77 | | |
| |||
123 | 121 | | |
124 | 122 | | |
125 | 123 | | |
| 124 | + | |
126 | 125 | | |
127 | 126 | | |
128 | 127 | | |
129 | | - | |
| 128 | + | |
130 | 129 | | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
135 | 135 | | |
136 | 136 | | |
137 | 137 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
904 | 904 | | |
905 | 905 | | |
906 | 906 | | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
907 | 910 | | |
908 | 911 | | |
909 | 912 | | |
910 | 913 | | |
911 | 914 | | |
912 | 915 | | |
913 | 916 | | |
914 | | - | |
915 | 917 | | |
916 | 918 | | |
917 | 919 | | |
| |||
0 commit comments