Skip to content

Commit 6cbc56a

Browse files
authored
fix(executorch): replay CUDA graphs on caller buffers and green contexts, order shared scratch calls with a GPU flag, check tensor types, and add an opt-in lean runtime build (#4983)
1 parent 99b565d commit 6cbc56a

28 files changed

Lines changed: 5375 additions & 638 deletions

‎cpp/BUILD‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -229,10 +229,10 @@ cc_library(
229229
strip_include_prefix = "src",
230230
)
231231

232-
# The one internal entry point of :tensorrt_executorch_backend that its test calls
233-
# directly. Under src/ for the same reason as the pool header above: it is an
234-
# implementation detail of execute(), and an installed header declaring it would
235-
# make it API from the next release.
232+
# Internal steps of :tensorrt_executorch_backend, split out so its test can call
233+
# install_pooled_scratch directly. Under src/ for the same reason as the pool
234+
# header above: they are implementation details of execute(), and an installed
235+
# header declaring them would make them API from the next release.
236236
#
237237
# It carries the definition and not only the declaration, so the target stands on
238238
# its own: a header target whose symbol another target defines links only for a

‎cpp/include/torch_tensorrt/executorch/TensorRTBackend.h‎

Lines changed: 32 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,14 @@ class TRTLogger : public nvinfer1::ILogger {
6262
* context's allocation strategy is fixed when the context is created, so a
6363
* later call governs only the methods loaded after it, and a pooled context and
6464
* a private-scratch one coexist in one process.
65+
*
66+
* The pool is one per device and belongs to the regular context of the first
67+
* pooled engine that runs there, so load every pooled engine on a device in
68+
* that context: one loaded in another regular context fails before it runs,
69+
* even on its own streams, with or without kCudaGraphsKey. The pool's context
70+
* must stay alive while the process uses the pool. With a CUDA 12.5 or newer
71+
* driver a pooled call on a stream of another regular context fails before it
72+
* sets its shapes or claims the pool.
6573
*/
6674
inline constexpr char kSharedActivationScratchKey[] = "use_shared_activation_scratch";
6775

@@ -90,13 +98,26 @@ inline constexpr char kSharedEnginesKey[] = "use_shared_engines";
9098
* b"0", overrides the process default, which is false unless set. Any other
9199
* compile spec value, or the key twice, fails the load. Engines with aliased
92100
* outputs, GPUs without stream-ordered memory, and drivers older than CUDA 12.5
93-
* keep ordinary enqueueV3. An engine on the shared activation scratch pool drops
94-
* its recorded graph whenever the pool hands it a different buffer, and pays the
95-
* same stable I/O buffers and copies as any other replaying engine. Caller
96-
* streams in the current ordinary context can replay, as can a call with no
97-
* caller stream. Green context and other context streams take the plain path,
98-
* and the graph is kept unless the scratch buffer, the shapes, or the
99-
* optimization profile changed.
101+
* keep ordinary enqueueV3. A replay reads and writes the buffers it was recorded
102+
* on, so keep each input and output in the same device buffer from call to call,
103+
* and replays copy nothing. A graph goes on the caller's buffers once a call
104+
* brings the buffers of the call before it, and on buffers of its own until
105+
* then. When the buffers under a graph on the caller's buffers move, that call
106+
* records again on the new ones. When they move a second time, for any shapes,
107+
* that graph records on buffers of its own and copies the inputs in and the
108+
* outputs out on every call, also after the shapes or the optimization profile
109+
* change. The plain graph and the graph of each green context count their own
110+
* moves. Host buffers on a GPU that does not read host memory directly, buffers
111+
* off a 256-byte boundary, inputs that are not row-major, and outputs that
112+
* share memory with an input are still copied on every call.
113+
* An engine on the shared activation scratch pool records again whenever the
114+
* pool hands it a different buffer. Streams of the current regular context,
115+
* green context streams of the engine's device, and calls with no caller stream
116+
* replay; the backend README covers the green context and other regular context
117+
* cases. The engine keeps the graphs it records on green contexts, so destroy
118+
* it before any green context it ran on. A stream of another device fails. An
119+
* engine on the shared activation scratch pool also fails on a stream of another
120+
* regular context; see kSharedActivationScratchKey.
100121
*/
101122
inline constexpr char kCudaGraphsKey[] = "use_cuda_graphs";
102123

@@ -135,9 +156,10 @@ class TensorRTBackend final : public ::executorch::runtime::BackendInterface {
135156
* a caller has to know:
136157
* - Calls on two such handles on one device are serialized at submission.
137158
* The backend does that itself, with a per-device lock held across the
138-
* enqueue. A handle loaded with the option off keeps its own scratch, and
139-
* so does one whose engine needs no activation scratch under any shape;
140-
* neither is affected.
159+
* enqueue, and on a GPU without 64-bit stream memory operations until the
160+
* call has finished. A handle loaded with the option off keeps its own
161+
* scratch, and so does one whose engine needs no activation scratch under
162+
* any shape; neither is affected.
141163
* - A call can wait on the host for pooled work submitted earlier on that
142164
* device, and a call that grows the pool can wait for an unbounded amount
143165
* of it. If the caller parks work that only the caller will release once

‎cpp/src/torch_tensorrt/executorch/EngineHandle.h‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
#include "torch_tensorrt/executorch/TensorRTBackend.h"
1818

1919
#include <NvInfer.h>
20+
#include <cuda.h>
2021

2122
#include <cstdint>
2223
#include <memory>
@@ -44,8 +45,23 @@ struct EngineHandle {
4445
std::vector<size_t> cached_input_sizes;
4546
std::vector<void*> cached_output_ptrs;
4647
std::vector<size_t> cached_output_sizes;
48+
// What exec_ctx holds, so execute() skips the TensorRT call when a shape or an address is the one it
49+
// already gave. Setting an unchanged input shape still costs about half a microsecond, and an address
50+
// or inferShapes tens of nanoseconds, and most callers pass the same ones every time. An unknown shape
51+
// has nbDims -1 and an unknown address is null. Only this delegate may set shapes, addresses or the
52+
// profile on exec_ctx, or these go stale.
53+
std::vector<nvinfer1::Dims> bound_input_dims;
54+
std::vector<void*> bound_addresses;
55+
// Changes whenever exec_ctx's input shapes or optimization profile may have changed, so a recorded
56+
// CUDA graph can tell that the shapes it was recorded under are gone.
57+
uint64_t shape_epoch = 0;
58+
// Whether inferShapes has succeeded since the shapes or the profile last changed.
59+
bool shapes_inferred = false;
4760
size_t num_inputs = 0;
4861
size_t num_outputs = 0;
62+
// The ExecuTorch type of each binding, inputs then outputs, read once at init so execute() checks each
63+
// tensor without a TensorRT call. Undefined where ExecuTorch has no such type, so no tensor matches it.
64+
std::vector<::executorch::aten::ScalarType> binding_types;
4965
// Per output binding [0..num_outputs): index into input_binding_names of the
5066
// input it aliases (in-place KV-cache / user alias), or -1 for a normal output.
5167
// Built at init from the blob's aliased_io. Either way execute() binds the
@@ -66,6 +82,16 @@ struct EngineHandle {
6682
std::vector<bool> input_is_alias_target;
6783
size_t num_aliased_outputs = 0;
6884
int device_id = 0;
85+
// The regular CUDA context current when exec_ctx was created. It owns the engine and the buffers execute()
86+
// keeps across calls; a call on a green context stream still runs in that green context. With a driver older
87+
// than CUDA 12.5, a green context current at load is recorded here instead.
88+
CUcontext context = nullptr;
89+
// That context's id, zero when the driver cannot report it. The driver can give a destroyed context's
90+
// handle to a new one, but never its id.
91+
unsigned long long context_id = 0;
92+
// The device's primary context, retained when a green context was current at load, so that context is
93+
// the one current while the engine and exec_ctx are created. Released when this handle is destroyed.
94+
CUcontext retained_primary = nullptr;
6995
// Whether this device can reach pageable host memory through shared host page tables, which is the
7096
// question the uses of this flag ask. A discrete card answers no: it can reach pageable memory, but
7197
// only by faulting pages in one at a time, so it takes the staged-copy path instead.
@@ -81,7 +107,8 @@ struct EngineHandle {
81107
// needing activation scratch under some shape; a handle loaded with the option
82108
// off keeps this false and says nothing about what its engine needs, because it
83109
// has its own kSTATIC scratch either way. It is a predicate and not a size
84-
// because the size is never the right amount to install: enqueueV3 refuses a
110+
// because the size is not the amount to install while TensorRT can size the
111+
// bound shapes (pooled_scratch_bytes_for_shapes): enqueueV3 refuses a
85112
// kUSER_MANAGED context with no device memory once the engine needs any, however
86113
// little the shapes bound to a given call need, so execute() must hand such a
87114
// call some buffer -- but the engine's own figure covers every shape in the
@@ -97,6 +124,16 @@ struct EngineHandle {
97124
// And a program gives an input the same dim order on every call.
98125
bool reordered_input_reported = false;
99126

127+
// setTensorAddress for binding `index`, inputs then outputs, unless exec_ctx already holds `address`.
128+
bool bind_address(size_t index, void* address);
129+
// setInputShape for input `index`, unless exec_ctx already holds `dims` and `again` is false.
130+
bool bind_input_shape(size_t index, const nvinfer1::Dims& dims, bool again = false);
131+
// setOptimizationProfileAsync, after forgetting every shape and address exec_ctx holds and the active profile.
132+
bool switch_profile(int32_t profile, cudaStream_t stream);
133+
// Clears the record of bound addresses. Call it when exec_ctx may hold other addresses, so the next
134+
// bind_address sets each.
135+
void forget_bound_addresses();
136+
100137
~EngineHandle();
101138
};
102139

0 commit comments

Comments
 (0)