Skip to content

Commit aa83485

Browse files
committed
Leave method inputs that stay on a device unplanned
With skip_h2d_for_method_inputs, a method input that a device delegate reads stays on the device, and the caller passes their own device tensor. Memory planning still reserved a buffer for that input by default. The runtime fills a planned input by copying into it, which it refuses to do for device memory, so the exported program failed on every call with "Planned-buffer copy needs host memory on both sides". MemoryPlanningPass(alloc_graph_input=False) avoided that, but it also unplans every host input of the method. PropagateDevicePass now marks such an input, as it marks constants that it keeps on the device, and memory planning never gives a marked input a buffer, whatever alloc_graph_input says. The method then reads the caller's tensor in place. Host inputs still follow alloc_graph_input, and the memory planning verifier leaves marked inputs out of its check. The PropagateDeviceConfig docstring and the CUDA and TensorRT pages no longer ask for alloc_graph_input=False. Tests: test_inputs_left_on_device_are_not_planned replaces a test that pinned the old default, and a CUDA test now runs such a program with the default planning. Both fail without the change. Assertions that expected a planned buffer for such an input are gone, since that plan fails on every call.
1 parent a45ddcd commit aa83485

7 files changed

Lines changed: 69 additions & 127 deletions

File tree

‎backends/cuda/tests/test_device_placement.py‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -124,8 +124,8 @@ def forward(self, x):
124124

125125
def test_input_on_device_also_read_on_host(self):
126126
"""With skip_h2d_for_method_inputs, the caller passes the input on the
127-
device. A delegate reads it there, and a host op reads a copy of it in
128-
host memory."""
127+
device, and the default memory planning leaves it unplanned. A delegate
128+
reads it there, and a host op reads a copy of it in host memory."""
129129

130130
class Model(torch.nn.Module):
131131
def forward(self, x, y):
@@ -136,7 +136,6 @@ def forward(self, x, y):
136136
skip_h2d_for_method_inputs=True
137137
),
138138
enable_non_cpu_memory_planning=True,
139-
memory_planning_pass=MemoryPlanningPass(alloc_graph_input=False),
140139
)
141140
model = Model()
142141
inputs = (torch.randn(2, 3), torch.randn(2, 3))

‎docs/source/backends/cuda/cuda-overview.md‎

Lines changed: 5 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -173,10 +173,8 @@ exec_program = et_program.to_executorch(
173173
skip_d2h_for_method_outputs=True,
174174
),
175175
enable_non_cpu_memory_planning=True,
176-
# Keeps the method from reserving its own input and output memory.
177-
memory_planning_pass=MemoryPlanningPass(
178-
alloc_graph_input=False, alloc_graph_output=False
179-
),
176+
# Keeps the method from reserving its own output memory.
177+
memory_planning_pass=MemoryPlanningPass(alloc_graph_output=False),
180178
)
181179
)
182180
```
@@ -201,10 +199,9 @@ it out works, and the snippet above sets it explicitly only to make the requirem
201199
Setting it to `False` while asking for any skip raises a `ValueError`, because copy insertion
202200
happens during device-aware memory planning.
203201

204-
**Inputs must be unplanned**, via `MemoryPlanningPass(alloc_graph_input=False)`. Without it the
205-
program reserves its own input buffer on the device, and the runtime refuses to copy the caller's
206-
tensor into it. The call fails with error `0x10` (`NotSupported`), and the log says
207-
`Planned-buffer copy needs host memory on both sides`.
202+
**Inputs that stay on the GPU are never planned.** The method reads the tensor you pass in place,
203+
whatever `alloc_graph_input` says, because the runtime cannot copy it into memory of its own. Other
204+
inputs still follow `alloc_graph_input`.
208205

209206
**Leave outputs unplanned too**, via `alloc_graph_output=False`. Then each result lands in memory
210207
you own, with no extra copy. A C++ caller must give each unplanned output a buffer once with

‎docs/source/backends/tensorrt/tensorrt-overview.md‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -197,18 +197,15 @@ torch_tensorrt.save(
197197
skip_h2d_for_method_inputs=True,
198198
skip_d2h_for_method_outputs=True,
199199
),
200-
memory_planning_pass=MemoryPlanningPass(
201-
alloc_graph_input=False, alloc_graph_output=False
202-
),
200+
memory_planning_pass=MemoryPlanningPass(alloc_graph_output=False),
203201
),
204202
)
205203
```
206204

207205
- The two skip flags work only with `enable_non_cpu_memory_planning`, which is on by default.
208206
With either flag set, turning it off makes the export raise a `ValueError`.
209-
- `alloc_graph_input=False` is required. Without it the program keeps its own input buffer, and
210-
every call fails with error `0x10` (`NotSupported`). The log then says
211-
`Planned-buffer copy needs host memory on both sides`.
207+
- Inputs that stay on the GPU are never planned: the method reads the tensor you pass in place,
208+
whatever `alloc_graph_input` says.
212209
- `alloc_graph_output=False` keeps the outputs out of the program's memory. From Python, the
213210
runtime allocates GPU memory for each output, and the next call does not overwrite an output you
214211
still hold, unless you pass `clone_outputs=False`. From C++, you must give the method your own

‎exir/memory_planning.py‎

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,7 @@ def verify_graph_input_output(self) -> None:
268268
mutable_buffer_specs = _get_mutable_buffer_specs(
269269
graph_module.graph.nodes, self.graph_signature
270270
)
271+
device_input_specs = _get_device_input_specs(graph_module.graph.nodes)
271272
for nd in graph_module.graph.nodes:
272273
if nd.op in check_list:
273274
if not (specs := get_node_tensor_specs(nd)):
@@ -277,13 +278,17 @@ def verify_graph_input_output(self) -> None:
277278
specs = list(
278279
filter(
279280
lambda spec: not spec.const
280-
and spec not in mutable_buffer_specs,
281+
and spec not in mutable_buffer_specs
282+
and spec not in device_input_specs,
281283
specs,
282284
)
283285
)
284286
if len(specs) == 0:
285-
# all outputs are const so no need to allocate memory just say we succeeded
286-
graph_output_allocated = self.alloc_graph_output
287+
# nothing left to allocate memory for, so just say we succeeded
288+
if nd.op == "placeholder":
289+
graph_input_allocated = self.alloc_graph_input
290+
else:
291+
graph_output_allocated = self.alloc_graph_output
287292
continue
288293
allocated = any(
289294
spec is None or spec.mem_offset is not None for spec in specs
@@ -397,6 +402,18 @@ def _is_device_resident_constant(node: Node) -> bool:
397402
return node.op == "placeholder" and node.meta.get("device_resident_constant", False)
398403

399404

405+
def _get_device_input_specs(nodes: Iterable[Node]) -> Set[TensorSpec]:
406+
"""Specs of the method inputs that PropagateDevicePass left on a device for
407+
skip_h2d_for_method_inputs. The runtime cannot copy the caller's tensor
408+
into planned device memory, so these always use the caller's memory."""
409+
return {
410+
spec
411+
for node in nodes
412+
if node.op == "placeholder" and node.meta.get("device_input", False)
413+
for spec in get_node_tensor_specs(node)
414+
}
415+
416+
400417
def _get_mutable_buffer_specs(
401418
nodes: Iterable[Node], graph_signature: Optional[ExportGraphSignature]
402419
) -> Set[TensorSpec]:
@@ -486,6 +503,7 @@ def collect_specs_from_nodes( # noqa: C901
486503
graph_output_tensors: Set[TensorSpec] = (
487504
get_graph_output_tensors(nodes) if ignore_graph_output else set()
488505
)
506+
device_input_tensors = _get_device_input_specs(nodes)
489507

490508
# Mutation outputs alias persistent state, whose allocation is controlled
491509
# by ignore_mutable_buffers independently of graph output allocation.
@@ -542,6 +560,8 @@ def collect_specs_from_nodes( # noqa: C901
542560
# we skip planning memory for graph input.
543561
if ignore_graph_input and spec in graph_input_tensors:
544562
continue
563+
if spec in device_input_tensors:
564+
continue
545565
if ignore_graph_output and spec in graph_output_tensors:
546566
continue
547567
if ignore_mutable_buffers and spec in mutable_buffer_specs:

‎exir/passes/propagate_device_config.py‎

Lines changed: 8 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -27,27 +27,22 @@
2727
class PropagateDeviceConfig:
2828
"""Controls whether the runtime copies method inputs, outputs and constants across the device boundary.
2929
30-
Memory planning reserves buffers for graph inputs and outputs by default. A program that skips
31-
the input copy must use ``alloc_graph_input=False``. Unplanning the outputs as well, as below,
32-
copies the least::
30+
Memory planning reserves buffers for graph inputs and outputs by default. An input whose copy
31+
is skipped is the exception: the runtime cannot copy the caller's tensor into device memory, so
32+
that input always uses the caller's memory, whatever ``alloc_graph_input`` says. Unplanning the
33+
outputs as well, as below, copies the least::
3334
3435
ExecutorchBackendConfig(
3536
propagate_device_config=PropagateDeviceConfig(
3637
skip_h2d_for_method_inputs=True,
3738
skip_d2h_for_method_outputs=True,
3839
),
3940
enable_non_cpu_memory_planning=True,
40-
memory_planning_pass=MemoryPlanningPass(
41-
alloc_graph_input=False, alloc_graph_output=False
42-
),
41+
memory_planning_pass=MemoryPlanningPass(alloc_graph_output=False),
4342
)
4443
45-
Without ``alloc_graph_input=False`` the runtime reserves device memory for an input the caller
46-
already owns and refuses to copy into it, so every call fails with ``NotSupported``. Planned
47-
outputs work, but they live in the method's memory, which the next run overwrites, and the
48-
Python ``execute()`` copies them unless it gets ``clone_outputs=False``.
49-
50-
``skip_h2d_for_constants`` does not need ``alloc_graph_input=False``: it works with planned inputs.
44+
Planned outputs work, but they live in the method's memory, which the next run overwrites, and
45+
the Python ``execute()`` copies them unless it gets ``clone_outputs=False``.
5146
"""
5247

5348
# When True, method-level input tensors that feed directly into a device
@@ -66,8 +61,7 @@ class PropagateDeviceConfig:
6661
# memory until the last delegate there has read it. Set
6762
# skip_h2d_for_constants to keep constants on the device instead.
6863
#
69-
# Pair with MemoryPlanningPass(alloc_graph_input=False), or the runtime reserves its own buffer
70-
# for the input and refuses the caller's tensor.
64+
# Memory planning gives such an input no buffer, so the method reads the caller's memory.
7165
# For per-method values, pass a dict of PropagateDeviceConfig, keyed by method name, as the
7266
# propagate_device_config of ExecutorchBackendConfig.
7367
skip_h2d_for_method_inputs: Union[bool, Dict[str, bool]] = False

‎exir/passes/propagate_device_pass.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -570,6 +570,9 @@ def _place_input_on_device(
570570
)
571571

572572
_set_device_on_spec(spec, device_type, device_index)
573+
# Memory planning leaves it unplanned, so the method uses the caller's
574+
# memory.
575+
placeholder.meta["device_input"] = True
573576
# With skip_d2h_for_method_outputs, the method returns the input where
574577
# the caller put it.
575578
other_readers = [

‎exir/tests/test_propagate_device_pass.py‎

Lines changed: 25 additions & 93 deletions
Original file line numberDiff line numberDiff line change
@@ -1653,7 +1653,7 @@ def forward(self, a, b):
16531653
enable_non_cpu_memory_planning=True,
16541654
)
16551655

1656-
for pipeline, program, gm in self._get_executorch_program(
1656+
for pipeline, _program, gm in self._get_executorch_program(
16571657
model, inputs, DeviceAwarePartitioner("cuda:0"), et_config
16581658
):
16591659
with self.subTest(pipeline=pipeline):
@@ -1683,16 +1683,6 @@ def forward(self, a, b):
16831683
expected_index=0,
16841684
)
16851685

1686-
# Verify buffer device mapping: CUDA placeholders should
1687-
# have their memory planned on a CUDA buffer.
1688-
self._assert_nodes_buffer_device(
1689-
cuda_ph,
1690-
program,
1691-
DeviceType.CUDA,
1692-
pipeline,
1693-
"Placeholder",
1694-
)
1695-
16961686
def test_skip_d2h_for_method_outputs(self):
16971687
"""When skip_d2h_for_method_outputs=True, delegate outputs that feed
16981688
directly to the graph output should NOT get _d2h_copy nodes."""
@@ -1808,15 +1798,6 @@ def forward(self, a, b):
18081798
"Delegate getitem",
18091799
)
18101800

1811-
# Verify buffer device mapping: both input and output
1812-
# buffers should be on CUDA.
1813-
self._assert_nodes_buffer_device(
1814-
cuda_ph,
1815-
program,
1816-
DeviceType.CUDA,
1817-
pipeline,
1818-
"Placeholder",
1819-
)
18201801
self._assert_nodes_buffer_device(
18211802
getitems,
18221803
program,
@@ -1876,16 +1857,9 @@ def forward(self, a, b):
18761857
f"device spec, got {len(cuda_ph)}",
18771858
)
18781859

1879-
# Verify buffer device mapping: the CUDA placeholder's
1880-
# buffer should be on CUDA, the CPU placeholder's buffer
1881-
# should be on CPU.
1882-
self._assert_nodes_buffer_device(
1883-
cuda_ph,
1884-
program,
1885-
DeviceType.CUDA,
1886-
pipeline,
1887-
"CUDA placeholder",
1888-
)
1860+
# The method reads the caller's memory for b, while a keeps
1861+
# its planned buffer on the CPU.
1862+
self.assertIsNone(cuda_ph[0].meta["spec"].mem_id)
18891863
self._assert_nodes_buffer_device(
18901864
cpu_ph,
18911865
program,
@@ -2386,7 +2360,7 @@ def forward(self, a, b):
23862360
),
23872361
enable_non_cpu_memory_planning=True,
23882362
)
2389-
for pipeline, program, gm in self._get_executorch_program(
2363+
for pipeline, _program, gm in self._get_executorch_program(
23902364
model, inputs, DeviceAwarePartitioner("cuda:0"), et_config
23912365
):
23922366
with self.subTest(pipeline=pipeline):
@@ -2396,9 +2370,6 @@ def forward(self, a, b):
23962370
self.assertEqual(len(nodes.h2d_nodes), 1)
23972371
cuda_ph, _ = self._collect_placeholders_by_device(gm)
23982372
self.assertEqual(sorted(n.name for n in cuda_ph), ["a", "b"])
2399-
self._assert_nodes_buffer_device(
2400-
cuda_ph, program, DeviceType.CUDA, pipeline, "Placeholder"
2401-
)
24022373

24032374
def test_skip_h2d_copies_input_for_readers_off_device(self):
24042375
"""With skip_h2d_for_method_inputs=True, a method input that a device
@@ -2634,74 +2605,35 @@ def test_tensorspec_repr_includes_device(self):
26342605
self.assertIn("device=", repr_str)
26352606
self.assertIn("CPU", repr_str)
26362607

2637-
def test_skipping_copies_requires_unallocated_graph_io(self):
2638-
"""Skipping a boundary copy is only complete when the runtime also stops reserving its own
2639-
buffer for that tensor.
2640-
2641-
Memory planning allocates graph inputs and outputs by default. A planned input carries
2642-
allocation info, and the runtime fills a planned input by copying the caller's memory into the
2643-
buffer it reserved, so the copy the export asked to skip comes back at run time. For device
2644-
memory that copy is also a host memcpy into a device pointer, which crashes.
2645-
2646-
This pins the pairing the configuration needs, because the two settings live in different
2647-
places and nothing else connects them.
2648-
"""
2608+
def test_inputs_left_on_device_are_not_planned(self):
2609+
"""The runtime fills a planned input by copying the caller's tensor into the buffer it
2610+
reserved, which it cannot do for device memory. So an input that skip_h2d_for_method_inputs
2611+
leaves on the device gets no buffer, whatever alloc_graph_input says, and the method reads
2612+
the caller's memory. The output still follows alloc_graph_output."""
26492613

26502614
class Model(torch.nn.Module):
26512615
def forward(self, a, b):
26522616
return torch.add(a, b)
26532617

2654-
model = Model()
2655-
inputs = (torch.randn(2, 2), torch.randn(2, 2))
2656-
skip_copies = PropagateDeviceConfig(
2657-
skip_h2d_for_method_inputs=True,
2658-
skip_d2h_for_method_outputs=True,
2659-
)
2660-
2661-
def planned_io(config: ExecutorchBackendConfig):
2662-
"""Whether the program reserves its own buffer for each graph input and output."""
2663-
lowered = to_edge_transform_and_lower(
2664-
torch.export.export(model, inputs),
2665-
partitioner=[DeviceAwarePartitioner("cuda:0")],
2666-
).to_executorch(config)
2667-
plan = lowered.executorch_program.execution_plan[0]
2668-
planned = lambda indices: [ # noqa: E731
2669-
getattr(plan.values[index].val, "allocation_info", None) is not None
2670-
for index in indices
2671-
]
2672-
return planned(plan.inputs), planned(plan.outputs)
2673-
2674-
# Default planning reserves buffers, so the runtime copies into them despite the skip flags.
2675-
planned_inputs, planned_outputs = planned_io(
2676-
ExecutorchBackendConfig(
2677-
propagate_device_config=skip_copies,
2678-
enable_non_cpu_memory_planning=True,
2679-
)
2680-
)
2681-
self.assertTrue(
2682-
all(planned_inputs),
2683-
"graph inputs are expected to be planned by default, which is why the pairing below "
2684-
f"is needed, but got {planned_inputs}",
2685-
)
2686-
2687-
# Asking planning to leave them alone is what actually removes the copy.
2688-
planned_inputs, planned_outputs = planned_io(
2618+
lowered = to_edge_transform_and_lower(
2619+
torch.export.export(Model(), (torch.randn(2, 2), torch.randn(2, 2))),
2620+
partitioner=[DeviceAwarePartitioner("cuda:0")],
2621+
).to_executorch(
26892622
ExecutorchBackendConfig(
2690-
propagate_device_config=skip_copies,
2691-
enable_non_cpu_memory_planning=True,
2692-
memory_planning_pass=MemoryPlanningPass(
2693-
alloc_graph_input=False, alloc_graph_output=False
2623+
propagate_device_config=PropagateDeviceConfig(
2624+
skip_h2d_for_method_inputs=True,
2625+
skip_d2h_for_method_outputs=True,
26942626
),
2627+
enable_non_cpu_memory_planning=True,
26952628
)
26962629
)
2697-
self.assertFalse(
2698-
any(planned_inputs),
2699-
f"no graph input should be planned when the caller provides it, got {planned_inputs}",
2700-
)
2701-
self.assertFalse(
2702-
any(planned_outputs),
2703-
f"no graph output should be planned when it stays on the device, got {planned_outputs}",
2704-
)
2630+
plan = lowered.executorch_program.execution_plan[0]
2631+
planned = lambda indices: [ # noqa: E731
2632+
getattr(plan.values[index].val, "allocation_info", None) is not None
2633+
for index in indices
2634+
]
2635+
self.assertEqual(planned(plan.inputs), [False, False])
2636+
self.assertEqual(planned(plan.outputs), [True])
27052637

27062638
def test_device_output_copied_to_host_is_planned(self):
27072639
"""A delegate output copied to the host is an intermediate tensor on the

0 commit comments

Comments
 (0)