Repository navigation
Commit 21c7c99
authored
CUDA backend: add a PTE device-id remap tool (#25315)
## Why
A CUDA program records the devices from its `target_device` compile spec
in three places in the PTE: the planned-memory table
(`non_const_buffer_device`), tensor `extra_tensor_info`, and each
delegate's `target_device` spec itself. All three must agree. A program
exported with an explicit non-zero `target_device` (`cuda:3`, say)
therefore does not load on a machine whose GPUs have other indices — the
loader tries to allocate planned buffers on a device that does not exist
(see #24887 discussion). Leaving `target_device` at
its `cuda:0` default avoids this; compiling on another GPU does not by
itself record that GPU.
Earlier I tried solving this at load time with a `target_device` runtime
option (#25310, closed). That spreads device knowledge into the generic
loaders, which only allocate ExecuTorch-owned buffers and should not
know any backend's options. This PR replaces that approach with an
offline tool, in the spirit of `merge_ptes.py`: rewrite the file once,
and the runtime runs it unchanged.
## What
`backends/cuda/scripts/remap_pte_devices.py`, which starts a new
`backends/cuda/scripts/` package for standalone CUDA backend tools
(`merge_ptes.py` moves into it in the stacked follow-up #25323):
```bash
python -m executorch.backends.cuda.scripts.remap_pte_devices \
--input-pte model.pte --output-pte model_dense.pte # 3,5,7 -> 0,1,2
python -m executorch.backends.cuda.scripts.remap_pte_devices \
--input-pte model.pte --output-pte model_cuda0.pte --mapping 3=0,5=0,7=0
python -m executorch.backends.cuda.scripts.remap_pte_devices \
--input-pte model.pte --dry-run # show the mapping only
```
- Collects every CUDA device index recorded in the program (planned
buffers, tensor infos, delegate `target_device` specs — parsed with
export's own `_parse_device_spec_value`, so `CUDA:3` and bare `cuda`
behave exactly as at export) and applies one mapping to all three
places, so they stay consistent. Collection and rewriting share one walk
over the device sites, deduplicated by object identity for specs several
delegates share.
- Default mapping compresses the recorded ids to a dense `0..n-1` range,
keeping them distinct and in order. The tool warns whenever the result
is not entirely on `cuda:0`: every remapped device must exist on the
target machine, and the CUDA runtime currently runs every delegate on
the current device (#24890). `--mapping OLD=NEW,...` overrides the
default and may collapse several devices onto `cuda:0`.
Distinct-preserving mappings load like a native export for the new
devices; collapsing mappings stay correct but keep device→host→device
bounces and per-device input copies a native single-device export would
have elided or shared.
- Refuses a mapping that does not cover a recorded device, names one the
program does not record, or names a target outside the stored byte range
`0..=127`, so a file is never rewritten only in part and a mapping meant
for another file fails loudly. Re-serialization uses
`extract_delegate_segments=True`, matching export; like merge_ptes it
re-serializes with the default alignments, so exports made with
non-default alignments may be repacked (contents unchanged). The
companion `.ptd` needs no changes: tensor data carries no device ids.
- No runtime changes. The CPU-only tests are wired into `pytest.ini`
testpaths and a Buck `python_unittest`, so serializer/schema changes run
them without a GPU.
## Tests
`backends/cuda/tests/test_remap_pte_devices.py` builds a program in
memory with devices spread across all three places and several execution
plans (planned buffers on cuda:3/cuda:5, a tensor on cuda:7, a delegate
spec on cuda:9, a second plan on cuda:11) and checks collection, dense
and explicit mappings (including collapsing to one device),
bare/uppercase/non-CUDA/malformed specs, shared-spec rewriting,
coverage/repeated-source/out-of-range errors, serialize round trips with
named data, identity-remap byte equality, failed-write atomicity, the
not-entirely-cuda:0 warning, the CLI dry-run (printed mapping), CLI
input errors as `SystemExit(2)`, and the CLI write path with
`--mapping`. It also exports a small program with
`DeviceAwarePartitioner("cuda:3")` and checks the remapped program
equals a native `cuda:0` export. All 22 tests pass locally (CPU-only
environment; no GPU needed since the tool only edits metadata).1 parent 3e8e3f8 commit 21c7c99
8 files changed
Lines changed: 947 additions & 0 deletions
File tree
- backends/cuda
- scripts
- tests
- examples/cuda
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
164 | 164 | | |
165 | 165 | | |
166 | 166 | | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
167 | 195 | | |
168 | 196 | | |
169 | 197 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
0 commit comments