Skip to content

Commit 21c7c99

Browse files
authored
CUDA backend: add a PTE device-id remap tool (#25315)
## Why A CUDA program records the devices from its `target_device` compile spec in three places in the PTE: the planned-memory table (`non_const_buffer_device`), tensor `extra_tensor_info`, and each delegate's `target_device` spec itself. All three must agree. A program exported with an explicit non-zero `target_device` (`cuda:3`, say) therefore does not load on a machine whose GPUs have other indices — the loader tries to allocate planned buffers on a device that does not exist (see #24887 discussion). Leaving `target_device` at its `cuda:0` default avoids this; compiling on another GPU does not by itself record that GPU. Earlier I tried solving this at load time with a `target_device` runtime option (#25310, closed). That spreads device knowledge into the generic loaders, which only allocate ExecuTorch-owned buffers and should not know any backend's options. This PR replaces that approach with an offline tool, in the spirit of `merge_ptes.py`: rewrite the file once, and the runtime runs it unchanged. ## What `backends/cuda/scripts/remap_pte_devices.py`, which starts a new `backends/cuda/scripts/` package for standalone CUDA backend tools (`merge_ptes.py` moves into it in the stacked follow-up #25323): ```bash python -m executorch.backends.cuda.scripts.remap_pte_devices \ --input-pte model.pte --output-pte model_dense.pte # 3,5,7 -> 0,1,2 python -m executorch.backends.cuda.scripts.remap_pte_devices \ --input-pte model.pte --output-pte model_cuda0.pte --mapping 3=0,5=0,7=0 python -m executorch.backends.cuda.scripts.remap_pte_devices \ --input-pte model.pte --dry-run # show the mapping only ``` - Collects every CUDA device index recorded in the program (planned buffers, tensor infos, delegate `target_device` specs — parsed with export's own `_parse_device_spec_value`, so `CUDA:3` and bare `cuda` behave exactly as at export) and applies one mapping to all three places, so they stay consistent. Collection and rewriting share one walk over the device sites, deduplicated by object identity for specs several delegates share. - Default mapping compresses the recorded ids to a dense `0..n-1` range, keeping them distinct and in order. The tool warns whenever the result is not entirely on `cuda:0`: every remapped device must exist on the target machine, and the CUDA runtime currently runs every delegate on the current device (#24890). `--mapping OLD=NEW,...` overrides the default and may collapse several devices onto `cuda:0`. Distinct-preserving mappings load like a native export for the new devices; collapsing mappings stay correct but keep device→host→device bounces and per-device input copies a native single-device export would have elided or shared. - Refuses a mapping that does not cover a recorded device, names one the program does not record, or names a target outside the stored byte range `0..=127`, so a file is never rewritten only in part and a mapping meant for another file fails loudly. Re-serialization uses `extract_delegate_segments=True`, matching export; like merge_ptes it re-serializes with the default alignments, so exports made with non-default alignments may be repacked (contents unchanged). The companion `.ptd` needs no changes: tensor data carries no device ids. - No runtime changes. The CPU-only tests are wired into `pytest.ini` testpaths and a Buck `python_unittest`, so serializer/schema changes run them without a GPU. ## Tests `backends/cuda/tests/test_remap_pte_devices.py` builds a program in memory with devices spread across all three places and several execution plans (planned buffers on cuda:3/cuda:5, a tensor on cuda:7, a delegate spec on cuda:9, a second plan on cuda:11) and checks collection, dense and explicit mappings (including collapsing to one device), bare/uppercase/non-CUDA/malformed specs, shared-spec rewriting, coverage/repeated-source/out-of-range errors, serialize round trips with named data, identity-remap byte equality, failed-write atomicity, the not-entirely-cuda:0 warning, the CLI dry-run (printed mapping), CLI input errors as `SystemExit(2)`, and the CLI write path with `--mapping`. It also exports a small program with `DeviceAwarePartitioner("cuda:3")` and checks the remapped program equals a native `cuda:0` export. All 22 tests pass locally (CPU-only environment; no GPU needed since the tool only edits metadata).
1 parent 3e8e3f8 commit 21c7c99

8 files changed

Lines changed: 947 additions & 0 deletions

File tree

‎backends/cuda/BUCK‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -164,6 +164,34 @@ fbcode_target(
164164
],
165165
)
166166

167+
fbcode_target(
168+
_kind = runtime.python_library,
169+
name = "scripts_atomic_output",
170+
srcs = [
171+
"scripts/__init__.py",
172+
"scripts/_atomic_output.py",
173+
],
174+
visibility = [
175+
"//executorch/backends/cuda/...",
176+
],
177+
)
178+
179+
fbcode_target(
180+
_kind = runtime.python_library,
181+
name = "remap_pte_devices",
182+
srcs = [
183+
"scripts/remap_pte_devices.py",
184+
],
185+
visibility = ["PUBLIC"],
186+
deps = [
187+
":scripts_atomic_output",
188+
"//executorch/exir/_serialize:lib",
189+
"//executorch/exir/backend:compile_spec_schema",
190+
"//executorch/exir/passes:propagate_device_pass",
191+
"//executorch/exir:schema",
192+
],
193+
)
194+
167195
fbcode_target(
168196
_kind = runtime.python_library,
169197
name = "cuda_partitioner",

‎backends/cuda/scripts/__init__.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
6+
7+
"""Write a CUDA tool's output file in one atomic step.
8+
9+
The tools in this package rewrite model files that other steps then
10+
load, so a half-written output must never be visible, a failed write
11+
must not leave its temp file behind, and the result must stay
12+
readable by whoever could read the input (a bare NamedTemporaryFile
13+
would leave it owner-only at 0600).
14+
"""
15+
16+
import os
17+
import stat
18+
import tempfile
19+
from pathlib import Path
20+
from typing import BinaryIO, Callable, Optional
21+
22+
23+
def write_atomically(
24+
output: Path,
25+
write: Callable[[BinaryIO], None],
26+
mode_from: Path,
27+
) -> None:
28+
"""Write `output` via a temp file in its directory, then replace it.
29+
30+
`write` receives the open temp file. The output takes
31+
`mode_from`'s permission bits (required, so no caller can fall
32+
back to the temp file's owner-only mode by omission).
33+
"""
34+
output.parent.mkdir(parents=True, exist_ok=True)
35+
temporary_path: Optional[Path] = None
36+
try:
37+
with tempfile.NamedTemporaryFile(
38+
dir=output.parent,
39+
prefix=f".{output.name}.",
40+
delete=False,
41+
) as temporary:
42+
temporary_path = Path(temporary.name)
43+
write(temporary)
44+
temporary.flush()
45+
os.fsync(temporary.fileno())
46+
os.chmod(temporary_path, stat.S_IMODE(mode_from.stat().st_mode))
47+
os.replace(temporary_path, output)
48+
temporary_path = None
49+
finally:
50+
if temporary_path is not None:
51+
temporary_path.unlink(missing_ok=True)

0 commit comments

Comments
 (0)