Skip to content

Commit c030572

Browse files
authored
Python runtime: enforce CUDA stream selection rules on the stream itself (#24327)
## What is wrong today `executorch.runtime.cuda` (added in #24273) lets Python code choose the CUDA stream that the CUDA and TensorRT delegates run on. Some rules were checked on what a wrapper says, not on the stream, and some docs promised too much: - A stream on GPU 1, given as `cuda.ExternalStream(ptr)` or `torch.cuda.ExternalStream(ptr)`, was accepted, then failed in the delegate. - `cudaStreamLegacy` (handle 1) got past the refusal of the legacy default stream (handle 0). - The docs said a closed stream cannot be selected, a dropped scope object ends its block, and a call inside a block waits only for its stream. None of these always holds. ## What this change does - In CUDA 13 builds, the native scope asks CUDA for the stream's device when a block starts (`cudaStreamGetDevice`) and refuses any device but 0. Every CUDA wheel built from main is CUDA 13. CUDA 12 builds keep only the wrapper check, since they can load a runtime older than 12.8, which lacks that function. On a stream that is capturing a CUDA graph, the query is skipped: CUDA refuses it there, and the refusal ends the caller's capture. - The native scope refuses handles 0 and 1, `None`, an exit from another thread, and a second entry. After refusing an exit out of order, it can be entered again once its block has ended. - `_CallerStreamScope.can_check_device` is a build-time attribute, not a CUDA call. - The docs say what is enforced, which waits a block shortens, and which steps still wait longer (outputs whose layout changes, loading CUDA weights). - New tests: a block inside `torch.cuda.graph()` leaves the graph intact, a refused native scope can be entered again, and CUDA 13 CI checks that the device query is built in. - `unittest-cuda` no longer starts for a change to `runtime/cuda.py` alone; nothing it runs imports it. ## Tests On Linux with A100 GPUs, bindings built with CUDA 13.2: - `test_pybindings_cuda_dlpack.py` passes in both output modes, with a second GPU so the device test runs. - Each new check fails without its fix: capture check removed, refused exit keeping its mark, device query compiled out, and the bindings from before these changes. - A library built against CUDA 12.8 that calls `cudaStreamGetDevice` fails to load next to the CUDA 12.6 runtime. Not run on Windows, ROCm, or a real CUDA 12 build.
1 parent 95d2d2b commit c030572

7 files changed

Lines changed: 286 additions & 71 deletions

File tree

‎.github/workflows/cuda.yml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -489,7 +489,6 @@ jobs:
489489
contains(needs.changed-files.outputs.changed-files, '.ci/scripts/test_model_e2e.sh') ||
490490
contains(needs.changed-files.outputs.changed-files, 'extension/pybindings') ||
491491
contains(needs.changed-files.outputs.changed-files, 'runtime/__init__.py') ||
492-
contains(needs.changed-files.outputs.changed-files, 'runtime/cuda.py') ||
493492
contains(needs.changed-files.outputs.changed-files, 'exir/passes/convert_constant_dim_order_pass.py') ||
494493
contains(needs.changed-files.outputs.changed-files, 'exir/passes/insert_write_back_for_buffers_pass.py') ||
495494
contains(needs.changed-files.outputs.changed-files, 'exir/passes/propagate_device_config.py') ||

‎extension/pybindings/README.md‎

Lines changed: 33 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -140,24 +140,39 @@ so while torch is loaded they still read CUDA inputs, torch tensors or DLPack,
140140
and return borrowed CUDA results.
141141

142142
Execution is synchronous at the Python boundary: a call returns after its device
143-
work has finished, so its outputs are ready on any stream. Every wait is for the
144-
whole device (`cudaDeviceSynchronize`), not for one stream, so it also waits for
145-
unrelated work queued on that device, for example by torch. Before it runs, a
146-
call waits once for each device that holds a non-empty CUDA input or output.
147-
Another library may still be writing such an input: a torch CUDA tensor, a CUDA
148-
DLPack input, or a result of these bindings passed back in, which can share
149-
memory a caller handed to that library. It may also still be reading the memory
150-
of an output from an earlier call, which this call writes again. After it runs,
151-
a call waits once for each device that holds a CUDA input or output, since the
152-
method may have updated an input in place. Each cloned CUDA
153-
output waits once more for its copy. A torch CUDA input in another layout also
154-
waits before torch copies it into the exported layout. Reading a CUDA result
155-
through `__dlpack__` or `to_host()` waits once as well. In CUDA-enabled
156-
bindings a wait holds the GIL, so other Python threads pause until it ends, and
157-
a queued CUDA host callback that needs the GIL makes the wait hang. Bindings
158-
built without CUDA wait through `torch.cuda.synchronize`, which releases the
159-
GIL. On an idle device a wait is short:
160-
`cudaDeviceSynchronize` took a median of 1.5 microseconds on an A100. The
143+
work has finished, so its outputs are ready on any stream. How long a call with
144+
only host inputs and outputs waits depends on where it runs:
145+
146+
- Inside an `executorch.runtime.cuda.stream()` block with a non-blocking stream,
147+
such as `executorch.runtime.cuda.Stream()`, it waits for that stream. A
148+
blocking stream, such as a `cupy.cuda.Stream()` with default flags, also waits
149+
for work on CUDA's legacy default stream, which is also the default stream of
150+
torch.
151+
- Outside a block, a program's copies between host and GPU memory run on the
152+
legacy default stream, so the call waits for work on every blocking stream of
153+
the device, for example delegates running on another thread.
154+
155+
In both cases, the CUDA delegate copies an output whose memory layout has to
156+
change through the legacy default stream, and loading a method's CUDA weights,
157+
which a first call may do, waits for the whole device.
158+
159+
The waits below, for CUDA inputs and outputs, are for the whole device
160+
(`cudaDeviceSynchronize`), not for one stream, so they also wait for unrelated
161+
work queued on that device, for example by torch. Before it runs, a call waits
162+
once for each device that holds a non-empty CUDA input or output. Another
163+
library may still be writing such an input: a torch CUDA tensor, a CUDA DLPack
164+
input, or a result of these bindings passed back in, which can share memory a
165+
caller handed to that library. It may also still be reading the memory of an
166+
output from an earlier call, which this call writes again. After it runs, a call
167+
waits once for each device that holds a CUDA input or output, since the method
168+
may have updated an input in place. Each cloned CUDA output waits once more for
169+
its copy. A torch CUDA input in another layout also waits before torch copies it
170+
into the exported layout. Reading a CUDA result through `__dlpack__` or
171+
`to_host()` waits once as well. In CUDA-enabled bindings a wait holds the GIL,
172+
so other Python threads pause until it ends, and a queued CUDA host callback
173+
that needs the GIL makes the wait hang. Bindings built without CUDA wait through
174+
`torch.cuda.synchronize`, which releases the GIL. On an idle device a wait is
175+
short: `cudaDeviceSynchronize` took a median of 1.5 microseconds on an A100. The
161176
synchronization has no C++ ATen dependency.
162177

163178
For bfloat16 inputs, raw byte buffers use the shape declared by the exported

‎extension/pybindings/pybindings.cpp‎

Lines changed: 84 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1818,19 +1818,47 @@ class PyCallerStreamScope final {
18181818
explicit PyCallerStreamScope(std::shared_ptr<PyCudaStream> owner)
18191819
: owner_(std::move(owner)) {}
18201820
explicit PyCallerStreamScope(uintptr_t stream)
1821-
: stream_(reinterpret_cast<cudaStream_t>(stream)) {}
1821+
: stream_(reinterpret_cast<cudaStream_t>(stream)) {
1822+
// 0 and cudaStreamLegacy (1) both name the legacy default stream, which
1823+
// waits for every blocking stream, so selecting it would defeat the point
1824+
// of choosing a stream. The CUDA delegate also refuses a null stream.
1825+
if (stream == 0 || stream == 1) {
1826+
throw std::invalid_argument(
1827+
"The legacy default CUDA stream (handle 0 or 1) cannot be selected; "
1828+
"select a stream created for this, such as cuda.Stream()");
1829+
}
1830+
}
18221831

1823-
// Losing the scope without an exit ends it at once, or, if blocks started
1824-
// after it are still active, when they end. Compared, never dereferenced:
1825-
// the starting thread may have ended.
1832+
// Losing the scope on the thread that entered it ends it at once, or when
1833+
// the blocks started after it end. Another thread cannot change the
1834+
// entering thread's selection, which then lasts until that thread ends.
1835+
// home_ is compared, never dereferenced: the entering thread may have ended.
18261836
~PyCallerStreamScope() {
18271837
if (id_ != 0 && home_ == &caller_stream_scopes) {
18281838
end(/*refuse_out_of_order=*/false);
18291839
}
18301840
}
18311841

1842+
// Whether enter() can check the device of a stream it did not create. Fixed
1843+
// when the bindings are built; see stream_device().
1844+
static constexpr bool kCanCheckDevice =
1845+
#if defined(CUDART_VERSION) && CUDART_VERSION >= 13000
1846+
true;
1847+
#else
1848+
false;
1849+
#endif
1850+
18321851
void enter() {
18331852
static std::atomic<uint64_t> next_id{1};
1853+
if (id_ != 0) {
1854+
throw std::runtime_error(
1855+
"This CUDA stream scope was entered and has not exited");
1856+
}
1857+
int device = 0;
1858+
if (!owner_ && stream_device(stream_, &device) && device != 0) {
1859+
throw std::runtime_error(
1860+
"Only a stream on CUDA device 0 can be selected for now");
1861+
}
18341862
const cudaStream_t stream = owner_ ? owner_->select() : stream_;
18351863
try {
18361864
auto guard =
@@ -1848,33 +1876,71 @@ class PyCallerStreamScope final {
18481876
}
18491877
}
18501878

1851-
// executorch.runtime.cuda enters each scope once, and exits it only on the
1852-
// thread that entered it.
18531879
void exit() {
1854-
if (id_ != 0) {
1855-
end(/*refuse_out_of_order=*/true);
1880+
if (id_ == 0) {
1881+
return;
1882+
}
1883+
if (home_ != &caller_stream_scopes) {
1884+
throw std::runtime_error(
1885+
"This CUDA stream scope was entered on another thread");
18561886
}
1887+
end(/*refuse_out_of_order=*/true);
18571888
}
18581889

18591890
private:
1891+
// Asks CUDA for the device of a stream rather than trusting the device a
1892+
// wrapper reports. Returns false when it cannot ask: in a CUDA 12 build, or
1893+
// while the stream captures a CUDA graph.
1894+
static bool stream_device(cudaStream_t stream, int* device) {
1895+
// cudaStreamGetDevice came in CUDA 12.8, but a CUDA 12 build may load any
1896+
// CUDA 12 runtime, where a missing symbol would fail the import. Every
1897+
// CUDA 13 runtime has it, and runs only on a driver that supports it.
1898+
#if defined(CUDART_VERSION) && CUDART_VERSION >= 13000
1899+
// CUDA refuses cudaStreamGetDevice on a capturing stream, and the refusal
1900+
// also ends the caller's capture.
1901+
cudaStreamCaptureStatus capture = cudaStreamCaptureStatusNone;
1902+
auto status = cudaStreamIsCapturing(stream, &capture);
1903+
if (status == cudaSuccess && capture != cudaStreamCaptureStatusNone) {
1904+
return false;
1905+
}
1906+
if (status == cudaSuccess) {
1907+
status = cudaStreamGetDevice(stream, device);
1908+
}
1909+
if (status != cudaSuccess) {
1910+
// A failed call leaves an error pending that the next CUDA call, even
1911+
// in another library, would report.
1912+
(void)cudaGetLastError();
1913+
throw std::runtime_error(
1914+
std::string("Failed to find the device of a CUDA stream: ") +
1915+
cudaGetErrorString(status));
1916+
}
1917+
return true;
1918+
#else
1919+
(void)stream;
1920+
(void)device;
1921+
return false;
1922+
#endif
1923+
}
1924+
18601925
void end(bool refuse_out_of_order) {
18611926
auto& scopes = caller_stream_scopes;
18621927
const auto it = std::find_if(
18631928
scopes.active.begin(), scopes.active.end(), [this](const auto& e) {
18641929
return e.id == id_;
18651930
});
18661931
if (it == scopes.active.end()) {
1867-
id_ = 0; // Already ended after an earlier refused exit.
1932+
id_ = 0; // Ended with the thread that entered it.
18681933
return;
18691934
}
18701935
if (it + 1 != scopes.active.end()) {
1936+
// Ends with the scopes above it, so this object can be entered again.
18711937
it->retired = true;
1938+
id_ = 0;
18721939
if (refuse_out_of_order) {
18731940
throw std::runtime_error(
18741941
"CUDA stream scopes must end innermost first; this one ends "
18751942
"when the scopes started after it end");
18761943
}
1877-
id_ = 0;
18781944
return;
18791945
}
18801946
scopes.pop_finished();
@@ -4840,8 +4906,15 @@ PYBIND11_MODULE(EXECUTORCH_PYTHON_MODULE_NAME, m) {
48404906
.def_property_readonly("device", &PyCudaStream::device)
48414907
.def("close", &PyCudaStream::close);
48424908
py::class_<PyCallerStreamScope>(m, "_CallerStreamScope", py::module_local())
4843-
.def(py::init<std::shared_ptr<PyCudaStream>>(), py::arg("stream"))
4909+
.def(
4910+
py::init<std::shared_ptr<PyCudaStream>>(),
4911+
py::arg("stream").none(false))
48444912
.def(py::init<uintptr_t>(), py::arg("stream"))
4913+
.def_property_readonly_static(
4914+
"can_check_device",
4915+
[](const py::object&) {
4916+
return PyCallerStreamScope::kCanCheckDevice;
4917+
})
48454918
.def("__enter__", &PyCallerStreamScope::enter)
48464919
.def(
48474920
"__exit__",

‎extension/pybindings/pybindings.pyi‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
from __future__ import annotations
99

1010
from enum import Enum
11-
from typing import Any, Callable, Dict, List, Optional, Tuple, TypeVar, Union
11+
from typing import Any, Callable, ClassVar, Dict, List, Optional, Tuple, TypeVar, Union
1212

1313
import torch
1414

@@ -497,5 +497,6 @@ class _CallerStreamScope:
497497
"""
498498

499499
def __init__(self, stream: Union[_CudaStream, int]) -> None: ...
500+
can_check_device: ClassVar[bool]
500501
def __enter__(self) -> None: ...
501502
def __exit__(self, *args: Any) -> None: ...

‎extension/pybindings/pybindings_cuda.h‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,8 +45,9 @@ inline std::shared_ptr<void> allocate_cuda_result(size_t nbytes, int index) {
4545
#endif
4646
}
4747

48-
// The Python API has no way to pass a stream or a readiness event, so the
49-
// Python boundary is synchronous and waits for the whole device.
48+
// A tensor handed across the Python boundary carries no readiness event, and
49+
// another library may write it on any stream, so the boundary waits for the
50+
// whole device, even inside an executorch.runtime.cuda stream block.
5051
inline void synchronize_cuda_device(int index) {
5152
#ifdef EXECUTORCH_PYBINDINGS_CUDA
5253
auto guard = executorch::extension::cuda::CUDAGuard::create(index);

0 commit comments

Comments
 (0)