Skip to content

Commit 9f49fe2

Browse files
authored
Fix main CUDA CI: oversized conv configs in the test, test-models-cuda timeout (#25392)
Fix the two CUDA jobs that fail on every `main` run since the #25270–#25280 stack landed: `unittest-cuda (kernel)` and `test-models-cuda`. That stack landed between `fe24cf5777` (both jobs green, 2026-11-19 06:48Z) and `6b82a162e7` (both red, 07:26Z). Every `main` run since then has failed the same way, most recently `e7f6aefd09`. ### `unittest-cuda (kernel)`: 5 `test_conv_implicit_gemm` failures All 5 failures are `OutOfResources: shared memory, Required: 131072, Hardware limit: 101376`: - `test_every_config_of_every_schedule` - `test_split_k` - `test_transposed` - `test_weight_stream` - `test_winograd` **The config that doesn't fit.** I compiled every `implicit_gemm.CONFIGS` entry for sm_86. Only one is too large for the A10G's 99 KiB: - It's the `igemm_tc_packed` tile `(128, 128, 64)` with 8 warps and 3 stages, in FP32 (TF32). That needs (3 − 1) × (128×64 + 64×128) × 4 B = 128 KiB. - It was added in `_TC_PACKED_TILES` by #25272. - The same tile in FP16/BF16 needs 64 KiB. **Why only the test fails.** - `check(every_config=True)` forces each config through `force_conv_candidate` and launches the raw JIT kernel, which bypasses the autotuner. - Each of the 5 tests has an FP32 every-config subtest with at least 16 input channels, which adds the packed schedule. - In production, Triton's autotuner times an `OutOfResources` config as `inf`, and Inductor's precompile drops it. So exports never select it. **Fix: the test, not the kernel.** - The test now skips a config that raises `OutOfResources` and emits a warning. This follows the existing pattern in `test_int8_quantized_gemm.py`. - It still fails if no config of a schedule fits. - Shrinking the tile to 2 stages would also change the FP16/BF16 tile that was tuned for A100/H100. ### `test-models-cuda`: cancelled at its 90-minute timeout Before the stack, the job took 64–70 minutes; since then it has been cancelled at 90.2 minutes on 11 runs in a row. Almost all of the extra time is in the conv-heavy exports: | step | `fe24cf5777` (good) | `6b82a162e7` (first bad) | |---|---|---| | resnet18 export | 27.9 min | 39.5 min | | mv2 export | 9.7 min | 22.3 min | | mv3 export | 9.2 min | cancelled after 9.6 min | **Why it got slower.** - #25278 turned on measured routing for the conv family. - Measured routing (#25277, #25279) is on by default. - So every convolution in resnet18, mv2 and mv3 is now compiled and timed in a worker process on a cold cache, against both Inductor and `triton::convolution`. This is the expected cost of that feature, and the job's job is to cover it, so I'm keeping it and raising the timeout to 120 minutes. `unittest-cuda` already uses the same limit. The projected runtime is about 100 minutes. ### Test plan - **Shared memory:** compiled every `implicit_gemm.CONFIGS` entry for sm_86 offline (triton 3.9). Only the FP32 `igemm_tc_packed (128, 128, 64, 8, 3)` needs more than 101376 B. - **New test logic, run without a GPU (with mocks):** - one oversized config: the test passes and warns `igemm_tc_packed big: shared memory`; - every config oversized: the test fails with `no igemm_tc_packed config fits this device`; - any other exception still propagates. - **Lint:** `black` 24.4.2 and `flake8` are clean, and `cuda.yml` parses. - **CI on this PR:** I'll confirm that `unittest-cuda (kernel)` has no `test_conv_implicit_gemm` failures and that `test-models-cuda` finishes within the new limit.
1 parent caf79c5 commit 9f49fe2

2 files changed

Lines changed: 49 additions & 6 deletions

File tree

‎.github/workflows/cuda.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -213,7 +213,9 @@ jobs:
213213
id-token: write
214214
contents: read
215215
with:
216-
timeout: 90
216+
# Measured conv routing times every convolution of resnet18, mv2 and mv3 at
217+
# export on a cold cache, which takes the job to about 100 minutes.
218+
timeout: 120
217219
runner: mt-l-x86aavx2-11-41-a10g
218220
gpu-arch-type: cuda
219221
gpu-arch-version: "13.2"

‎backends/cuda/tests/test_conv_implicit_gemm.py‎

Lines changed: 46 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
import os
1515
import tempfile
1616
import unittest
17+
import warnings
1718
from types import SimpleNamespace
1819
from unittest import mock
1920

@@ -69,6 +70,22 @@ def _channels_last(t):
6970
)
7071

7172

73+
# (schedule, (BLOCK_M, BLOCK_N, BLOCK_K, num_warps, num_stages)) tiles that need more
74+
# shared memory in FP32 than a 99 KiB GPU (A10G, sm_86) has.
75+
_OVERSIZED_FP32_TILES = {("igemm_tc_packed", (128, 128, 64, 8, 3))}
76+
77+
78+
def _tile(config):
79+
kwargs = config.kwargs
80+
return (
81+
kwargs["BLOCK_M"],
82+
kwargs["BLOCK_N"],
83+
kwargs["BLOCK_K"],
84+
config.num_warps,
85+
config.num_stages,
86+
)
87+
88+
7289
class ImplicitGemmScheduleTest(unittest.TestCase):
7390
def setUp(self):
7491
if not torch.cuda.is_available() or torch.version.hip is not None:
@@ -121,7 +138,8 @@ def check(
121138
each or every config, and checks the result and the layout (the fake
122139
implementation's, and ``output_strides`` if given); the tensor-core
123140
schedule runs FP32 operands in TF32. Transposed if ``output_padding`` is
124-
given."""
141+
given. With every config, the one tile known to need more shared memory
142+
than the device has is skipped with a warning."""
125143
from executorch.backends.cuda.triton.kernels.conv.conv import (
126144
force_conv_candidate,
127145
)
@@ -135,6 +153,7 @@ def check(
135153
ConvParams,
136154
out_shape_and_stride,
137155
)
156+
from triton.runtime.errors import OutOfResources
138157

139158
rank = x.dim() - 2
140159
params = ConvParams(
@@ -190,11 +209,33 @@ def check(
190209
and x.dtype == torch.float32
191210
):
192211
tolerance = _TF32_TOLERANCE
212+
checked = 0
193213
for config in configs if every_config else configs[:1]:
194-
with tf32, force_conv_candidate(schedule, config):
195-
out = self.convolution(
196-
x, w, b, stride, before, after, output_strides, output_padding
197-
)
214+
try:
215+
with tf32, force_conv_candidate(schedule, config):
216+
out = self.convolution(
217+
x,
218+
w,
219+
b,
220+
stride,
221+
before,
222+
after,
223+
output_strides,
224+
output_padding,
225+
)
226+
except OutOfResources as e:
227+
# The autotuner drops a config that does not fit the device. Only
228+
# this tile is known not to fit: in FP32 it needs 128 KiB, and an
229+
# A10G has 99 KiB. Any other config that runs out is a regression.
230+
if not (
231+
every_config
232+
and x.dtype == torch.float32
233+
and (schedule, _tile(config)) in _OVERSIZED_FP32_TILES
234+
):
235+
raise
236+
warnings.warn(f"{schedule} {config}: {e}", stacklevel=1)
237+
continue
238+
checked += 1
198239
self.assertEqual((tuple(out.shape), out.stride()), layout)
199240
torch.testing.assert_close(
200241
out.double(),

0 commit comments

Comments
 (0)