Repository navigation
Commit 9f49fe2
authored
Fix main CUDA CI: oversized conv configs in the test, test-models-cuda timeout (#25392)
Fix the two CUDA jobs that fail on every `main` run since the
#25270–#25280 stack landed: `unittest-cuda (kernel)` and
`test-models-cuda`. That stack landed between `fe24cf5777` (both jobs
green, 2026-11-19 06:48Z) and `6b82a162e7` (both red, 07:26Z). Every
`main` run since then has failed the same way, most recently
`e7f6aefd09`.
### `unittest-cuda (kernel)`: 5 `test_conv_implicit_gemm` failures
All 5 failures are `OutOfResources: shared memory, Required: 131072,
Hardware limit: 101376`:
- `test_every_config_of_every_schedule`
- `test_split_k`
- `test_transposed`
- `test_weight_stream`
- `test_winograd`
**The config that doesn't fit.** I compiled every
`implicit_gemm.CONFIGS` entry for sm_86. Only one is too large for the
A10G's 99 KiB:
- It's the `igemm_tc_packed` tile `(128, 128, 64)` with 8 warps and 3
stages, in FP32 (TF32). That needs (3 − 1) × (128×64 + 64×128) × 4 B =
128 KiB.
- It was added in `_TC_PACKED_TILES` by #25272.
- The same tile in FP16/BF16 needs 64 KiB.
**Why only the test fails.**
- `check(every_config=True)` forces each config through
`force_conv_candidate` and launches the raw JIT kernel, which bypasses
the autotuner.
- Each of the 5 tests has an FP32 every-config subtest with at least 16
input channels, which adds the packed schedule.
- In production, Triton's autotuner times an `OutOfResources` config as
`inf`, and Inductor's precompile drops it. So exports never select it.
**Fix: the test, not the kernel.**
- The test now skips a config that raises `OutOfResources` and emits a
warning. This follows the existing pattern in
`test_int8_quantized_gemm.py`.
- It still fails if no config of a schedule fits.
- Shrinking the tile to 2 stages would also change the FP16/BF16 tile
that was tuned for A100/H100.
### `test-models-cuda`: cancelled at its 90-minute timeout
Before the stack, the job took 64–70 minutes; since then it has been
cancelled at 90.2 minutes on 11 runs in a row. Almost all of the extra
time is in the conv-heavy exports:
| step | `fe24cf5777` (good) | `6b82a162e7` (first bad) |
|---|---|---|
| resnet18 export | 27.9 min | 39.5 min |
| mv2 export | 9.7 min | 22.3 min |
| mv3 export | 9.2 min | cancelled after 9.6 min |
**Why it got slower.**
- #25278 turned on measured routing for the conv family.
- Measured routing (#25277, #25279) is on by default.
- So every convolution in resnet18, mv2 and mv3 is now compiled and
timed in a worker process on a cold cache, against both Inductor and
`triton::convolution`.
This is the expected cost of that feature, and the job's job is to cover
it, so I'm keeping it and raising the timeout to 120 minutes.
`unittest-cuda` already uses the same limit. The projected runtime is
about 100 minutes.
### Test plan
- **Shared memory:** compiled every `implicit_gemm.CONFIGS` entry for
sm_86 offline (triton 3.9). Only the FP32 `igemm_tc_packed (128, 128,
64, 8, 3)` needs more than 101376 B.
- **New test logic, run without a GPU (with mocks):**
- one oversized config: the test passes and warns `igemm_tc_packed big:
shared memory`;
- every config oversized: the test fails with `no igemm_tc_packed config
fits this device`;
- any other exception still propagates.
- **Lint:** `black` 24.4.2 and `flake8` are clean, and `cuda.yml`
parses.
- **CI on this PR:** I'll confirm that `unittest-cuda (kernel)` has no
`test_conv_implicit_gemm` failures and that `test-models-cuda` finishes
within the new limit.1 parent caf79c5 commit 9f49fe2
2 files changed
Lines changed: 49 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
213 | 213 | | |
214 | 214 | | |
215 | 215 | | |
216 | | - | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
217 | 219 | | |
218 | 220 | | |
219 | 221 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
| |||
69 | 70 | | |
70 | 71 | | |
71 | 72 | | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
72 | 89 | | |
73 | 90 | | |
74 | 91 | | |
| |||
121 | 138 | | |
122 | 139 | | |
123 | 140 | | |
124 | | - | |
| 141 | + | |
| 142 | + | |
125 | 143 | | |
126 | 144 | | |
127 | 145 | | |
| |||
135 | 153 | | |
136 | 154 | | |
137 | 155 | | |
| 156 | + | |
138 | 157 | | |
139 | 158 | | |
140 | 159 | | |
| |||
190 | 209 | | |
191 | 210 | | |
192 | 211 | | |
| 212 | + | |
193 | 213 | | |
194 | | - | |
195 | | - | |
196 | | - | |
197 | | - | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
198 | 239 | | |
199 | 240 | | |
200 | 241 | | |
| |||
0 commit comments