Skip to content

Commit 306a9de

Browse files
committed
fix(executorch): move compile_and_save's engines into the program instead of copying them
`compile_and_save` writes each engine to a temporary file as soon as it is built. The ExecuTorch save then copies that file into the `.pte` (or the `.ptd`), and the file is deleted only when the call returns. So while the program is written, every engine is on disk twice. For a model whose engine is tens of gigabytes, the copy needs that much more free disk, and an export can fail on a device that has room for the result. The backend now hands each of these engine files to ExecuTorch as data that owns the file (`FileBackedData(path, cleanup=True)`); it is deleted after the save anyway. Inside `compile_and_save`, the save then writes the program and the `.ptd` with `consume=True`, so ExecuTorch moves each engine into the program: it copies it back to front and truncates the file as it goes, and each engine is on disk once. ExecuTorch does this only when the temporary directory, where the engine files are, is on the same file system as the program; otherwise it copies. With `generate_etrecord`, the save copies the engines instead. The ETRecord reads the engines after the program is written, so the save order stays the same as before, and the program is kept when the ETRecord can't be written. Every ExecuTorch save now passes `consume` (as `False` outside `compile_and_save`), so this needs the `consume` argument from pytorch/executorch#25099. It lands after that change, together with an ExecuTorch pin that has it, which also raises the minimum ExecuTorch version. Test plan: - `tests/py/dynamo/executorch/test_api.py`, CPU, with the lowering stubbed: - New `test_compile_and_save_moves_the_engines`: inside `compile_and_save`'s offload block, the program and the `.ptd` are written with `consume=True`. It fails without this change. - New `test_compile_and_save_keeps_the_program_when_the_etrecord_fails`: in the same block with an ETRecord, the program and the `.ptd` are written first, without `consume`, and the `.pte` is kept when the ETRecord save raises. - New `test_save_executorch_keeps_the_program_when_the_etrecord_fails`: the same for a plain save. - New `test_save_executorch_does_not_consume`: a plain save writes without `consume`. - `tests/py/dynamo/executorch/test_compile_and_save.py` on an A100: - The streaming test now also checks that each engine reaches ExecuTorch as data that owns its file. - New `test_engines_move_into_the_program_without_a_second_copy` checks that each engine file is empty after the program is written. - New `test_an_etrecord_makes_the_save_copy_the_engines`: with an ETRecord, the engine files keep their bytes, and both the `.pte` and the ETRecord are written.
1 parent 14a181c commit 306a9de

7 files changed

Lines changed: 174 additions & 11 deletions

File tree

‎docsrc/user_guide/runtime_performance/saving_models.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -281,7 +281,7 @@ warning and are ignored. With this format, an unknown keyword raises a ``TypeErr
281281
:ref:`Zero-copy KV cache <executorch_zero_copy_kv>`.
282282
* ``generate_etrecord``: also write an ETRecord, a debug record for the ExecuTorch Developer Tools
283283
``Inspector``, as ``<name>_etrecord.bin``. It does not work for a program with TensorRT
284-
engines: writing the record fails with a ``SerializeError``.
284+
engines: writing the record fails with a ``SerializeError``. ``compile_and_save`` can write one.
285285

286286
:ref:`executorch_deployment` explains the run time side of ``use_cuda_graphs``,
287287
``weight_streaming_budget_per_engine`` and ``external_engine_data``.

‎py/torch_tensorrt/_compile.py‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@
7777
from torch_tensorrt.dynamo._tracer import trace as dynamo_trace
7878
from torch_tensorrt.dynamo.runtime import TorchTensorRTModule
7979
from torch_tensorrt.dynamo.runtime._offloaded_engines import (
80+
defer_engine_setup_to_load,
8081
offload_engines_to_disk,
8182
)
8283
from torch_tensorrt.dynamo.utils import get_torch_inputs
@@ -873,6 +874,7 @@ def save(
873874
``<model>_etrecord.bin`` next to the ``.pte`` for use with the
874875
ExecuTorch Inspector. It does not work for a program with
875876
TensorRT engines: writing the record raises a ``SerializeError``.
877+
``compile_and_save`` can write one.
876878
``weight_streaming_budget_per_engine=`` takes an ``Optional[int]``
877879
number of bytes of engine weights that may stay resident in GPU
878880
memory, with the rest streamed from host memory. It applies to
@@ -1550,6 +1552,10 @@ def compile_and_save(
15501552
but each engine is written to a file as soon as it is built, so no built engine is held on the GPU
15511553
or in host memory while the next ones build. With ``output_format="executorch"`` no engine is ever
15521554
loaded: the save streams each file into the ``.pte`` (or the ``.ptd`` with ``external_engine_data``).
1555+
Without ``generate_etrecord``, the save moves each file there instead, so no engine is on disk twice,
1556+
when the temporary directory (``TMPDIR``) is on the same file system as ``file_path``. With
1557+
``generate_etrecord``, the save copies the engines and writes the ETRecord after the program, so the
1558+
program is kept if the ETRecord can't be written.
15531559
The other formats save each engine from a loaded one, so their engines are loaded once all of them
15541560
are built, and their save needs as much memory as after ``compile``.
15551561
@@ -1600,7 +1606,9 @@ def compile_and_save(
16001606
save(trt_gm, os.fspath(file_path), output_format=output_format, **save_kwargs)
16011607

16021608

1603-
def _write_external_tensor_data(executorch_program: Any, file_path: str) -> None:
1609+
def _write_external_tensor_data(
1610+
executorch_program: Any, file_path: str, consume: bool = False
1611+
) -> None:
16041612
"""Write an ExecuTorch program's external named tensor data (``.ptd``) next to the ``.pte``.
16051613
16061614
The CUDA (AOTInductor) backend emits its weights as external named data
@@ -1616,7 +1624,7 @@ def _write_external_tensor_data(executorch_program: Any, file_path: str) -> None
16161624
# Direct attribute access (no getattr default) so a future rename fails loud.
16171625
if executorch_program._tensor_data:
16181626
out_dir = os.path.dirname(os.path.abspath(file_path))
1619-
executorch_program.write_tensor_data_to_file(out_dir)
1627+
executorch_program.write_tensor_data_to_file(out_dir, consume=consume)
16201628
# write_tensor_data_to_file adds .ptd unless the name already ends with it.
16211629
paths = [
16221630
os.path.join(out_dir, name if name.endswith(".ptd") else f"{name}.ptd")
@@ -1752,9 +1760,13 @@ def _lower_and_write_executorch(
17521760
# the same call is the caller's to make, because finalization happens on
17531761
# ExecuTorch's own manager and out of this library's sight.
17541762
check_zero_copy_kv(executorch_program)
1763+
# Inside compile_and_save the engine files are deleted after this save, so it moves them into
1764+
# the program. The ETRecord reads the engines after the program is written, so with one the save
1765+
# copies them, which also keeps the program when the ETRecord can't be saved.
1766+
consume = defer_engine_setup_to_load() and not generate_etrecord
17551767
with open(file_path, "wb") as f:
1756-
executorch_program.write_to_file(f)
1757-
_write_external_tensor_data(executorch_program, file_path)
1768+
executorch_program.write_to_file(f, consume=consume)
1769+
_write_external_tensor_data(executorch_program, file_path, consume)
17581770
if generate_etrecord:
17591771
# Follows ExecuTorch's example convention (e.g. examples/cuda/scripts/export.py):
17601772
# persist the ETRecord as "<pte_base>_etrecord.bin" next to the .pte.

‎py/torch_tensorrt/dynamo/runtime/_offloaded_engines.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
engines built so far are then held neither on the GPU nor in host memory while the next ones build.
99
``compile_module`` leaves them in their files: an ExecuTorch save streams each file into the program, and
1010
``compile_and_save`` loads them before any other save. The files are deleted when the block ends.
11+
A file the save moved stays mapped, empty, until then, so nothing may read its engine after the save.
1112
"""
1213

1314
import contextlib

‎py/torch_tensorrt/executorch/_export.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -599,6 +599,7 @@ def export(
599599
generate_etrecord (bool): Ask ExecuTorch for an ETRecord for later debugging.
600600
This copies the whole program, engines included. Saving the record does not
601601
work for a program with TensorRT engines: it raises a ``SerializeError``.
602+
``torch_tensorrt.compile_and_save`` can write one.
602603
weight_streaming_budget_per_engine (Optional[int]): Bytes of engine weights that
603604
may stay resident in GPU memory, with the rest streamed from host memory.
604605
It applies to **each** TensorRT engine separately, not as a total for the

‎py/torch_tensorrt/executorch/backend.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -699,7 +699,9 @@ def preprocess(
699699

700700
engine_data: Any
701701
if engine_file is not None:
702-
engine_data = FileBackedData(engine_file)
702+
# compile_and_save deletes the file after the save, so the data can own it, which lets
703+
# a consuming write move the engine into the program instead of copying it.
704+
engine_data = FileBackedData(engine_file, cleanup=True)
703705
engine_data_sha256 = engine_data.sha256()
704706
else:
705707
assert isinstance(engine_bytes, bytes)

‎tests/py/dynamo/executorch/test_api.py‎

Lines changed: 92 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2824,28 +2824,42 @@ def test_lift_keeps_parameter_for_any_dtype(dtype, requires_grad):
28242824
# next to the .pte.
28252825

28262826

2827-
def _patch_executorch_lowering(monkeypatch, captured):
2827+
def _patch_executorch_lowering(monkeypatch, captured, log=None, etrecord_error=None):
28282828
"""Spy the seams _save_as_executorch delegates to after #4440, without running a
28292829
real TRT lowering. Records the kwargs forwarded into
28302830
torch_tensorrt.executorch.export() (captured["export_kwargs"]); lets the real
28312831
export() run against an engine-free program while stubbing only the innermost
28322832
ExecuTorch lowering, so the kwargs export() finally hands that lowering --
28332833
including the get_edge_compile_config() it substitutes for an omitted
28342834
compile_config -- land in `captured`; and records backend_config from
2835-
to_executorch(). Fills `captured`; returns nothing."""
2835+
to_executorch(). Fills `captured`; returns nothing. With `log`, records the order of
2836+
the ETRecord, program and .ptd writes, and whether each write consumes;
2837+
`etrecord_error` makes the ETRecord save raise."""
28362838
import executorch.exir as exir
28372839
import torch_tensorrt._compile as tc
28382840
import torch_tensorrt.executorch as tte
28392841

28402842
class _FakeETRecord:
28412843
def save(self, path):
2844+
if log is not None:
2845+
log.append("etrecord")
2846+
if etrecord_error is not None:
2847+
raise etrecord_error
28422848
with open(path, "wb") as fh:
28432849
fh.write(b"etrecord")
28442850

28452851
class _FakeExec:
2846-
def write_to_file(self, f):
2852+
_tensor_data = {"model": b""}
2853+
2854+
def write_to_file(self, f, *, consume=False):
2855+
if log is not None:
2856+
log.append(("program", consume))
28472857
f.write(b"")
28482858

2859+
def write_tensor_data_to_file(self, outdir, *, consume=False):
2860+
if log is not None:
2861+
log.append(("ptd", consume))
2862+
28492863
def get_etrecord(self):
28502864
return _FakeETRecord()
28512865

@@ -2875,7 +2889,6 @@ def _spy_export(source, **kw):
28752889
return real_export(source, **kw)
28762890

28772891
monkeypatch.setattr(tte, "export", _spy_export)
2878-
monkeypatch.setattr(tc, "_write_external_tensor_data", lambda prog, path: None)
28792892
# ENABLED_FEATURES is an immutable namedtuple; swap the whole module attribute.
28802893
monkeypatch.setattr(
28812894
tc, "ENABLED_FEATURES", types.SimpleNamespace(torch_tensorrt_runtime=True)
@@ -2933,6 +2946,81 @@ def test_save_executorch_forwards_lowering_kwargs(monkeypatch, tmp_path):
29332946
assert (tmp_path / "model_etrecord.bin").exists()
29342947

29352948

2949+
@pytest.mark.unit
2950+
def test_save_executorch_keeps_the_program_when_the_etrecord_fails(
2951+
monkeypatch, tmp_path
2952+
):
2953+
pytest.importorskip("executorch.exir")
2954+
import torch_tensorrt._compile as tc
2955+
2956+
log = []
2957+
_patch_executorch_lowering(
2958+
monkeypatch, {}, log=log, etrecord_error=RuntimeError("cannot serialize")
2959+
)
2960+
out = tmp_path / "model.pte"
2961+
ep = torch.export.export(_AddOne(), (torch.randn(2, 2),))
2962+
with pytest.raises(RuntimeError, match="cannot serialize"):
2963+
tc._save_as_executorch(ep, str(out), generate_etrecord=True)
2964+
2965+
assert log == [("program", False), ("ptd", False), "etrecord"]
2966+
assert out.exists()
2967+
2968+
2969+
@pytest.mark.unit
2970+
def test_save_executorch_does_not_consume(monkeypatch, tmp_path):
2971+
pytest.importorskip("executorch.exir")
2972+
import torch_tensorrt._compile as tc
2973+
2974+
log = []
2975+
_patch_executorch_lowering(monkeypatch, {}, log=log)
2976+
ep = torch.export.export(_AddOne(), (torch.randn(2, 2),))
2977+
tc._save_as_executorch(ep, str(tmp_path / "model.pte"))
2978+
2979+
assert log == [("program", False), ("ptd", False)]
2980+
2981+
2982+
@pytest.mark.unit
2983+
def test_compile_and_save_moves_the_engines(monkeypatch, tmp_path):
2984+
pytest.importorskip("executorch.exir")
2985+
import torch_tensorrt._compile as tc
2986+
from torch_tensorrt.dynamo.runtime._offloaded_engines import (
2987+
offload_engines_to_disk,
2988+
)
2989+
2990+
log = []
2991+
_patch_executorch_lowering(monkeypatch, {}, log=log)
2992+
ep = torch.export.export(_AddOne(), (torch.randn(2, 2),))
2993+
# The block compile_and_save saves in, where the engine files are deleted after the save.
2994+
with offload_engines_to_disk():
2995+
tc._save_as_executorch(ep, str(tmp_path / "model.pte"))
2996+
2997+
assert log == [("program", True), ("ptd", True)]
2998+
2999+
3000+
@pytest.mark.unit
3001+
def test_compile_and_save_keeps_the_program_when_the_etrecord_fails(
3002+
monkeypatch, tmp_path
3003+
):
3004+
pytest.importorskip("executorch.exir")
3005+
import torch_tensorrt._compile as tc
3006+
from torch_tensorrt.dynamo.runtime._offloaded_engines import (
3007+
offload_engines_to_disk,
3008+
)
3009+
3010+
log = []
3011+
_patch_executorch_lowering(
3012+
monkeypatch, {}, log=log, etrecord_error=OSError("No space left on device")
3013+
)
3014+
out = tmp_path / "model.pte"
3015+
ep = torch.export.export(_AddOne(), (torch.randn(2, 2),))
3016+
with offload_engines_to_disk():
3017+
with pytest.raises(OSError, match="No space left"):
3018+
tc._save_as_executorch(ep, str(out), generate_etrecord=True)
3019+
3020+
assert log == [("program", False), ("ptd", False), "etrecord"]
3021+
assert out.exists()
3022+
3023+
29363024
@pytest.mark.unit
29373025
def test_save_executorch_defaults_when_lowering_kwargs_omitted(monkeypatch, tmp_path):
29383026
pytest.importorskip("executorch.exir")

‎tests/py/dynamo/executorch/test_compile_and_save.py‎

Lines changed: 60 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ def forward(self, x):
3131
@pytest.mark.unit
3232
def test_engines_stream_from_their_files_into_the_program(tmp_path, monkeypatch):
3333
monkeypatch.setattr(tempfile, "tempdir", str(tmp_path))
34-
setups, streamed = [], {}
34+
setups, streamed, owned = [], {}, []
3535
real_setup = TorchTensorRTModule.setup_engine
3636
real_add = NamedDataStore.add_named_data
3737

@@ -43,6 +43,8 @@ def recording_add(self, key, data, *args, **kwargs):
4343
if isinstance(data, FileBackedData):
4444
with open(data._path, "rb") as f:
4545
streamed[data._path] = f.read()
46+
# Data that owns its file is what a consuming save may move.
47+
owned.append(data._finalizer is not None)
4648
else:
4749
streamed[key] = data
4850
return real_add(self, key, data, *args, **kwargs)
@@ -65,6 +67,7 @@ def recording_add(self, key, data, *args, **kwargs):
6567
# No engine was ever set up, and each one reached the program as its own file, from the cache directory.
6668
assert setups == []
6769
assert len(streamed) == 2
70+
assert owned and all(owned)
6871
offloaded = str(_cache_root() / "offloaded_engines")
6972
assert all(
7073
path.endswith(".engine") and os.path.dirname(path) == offloaded
@@ -74,3 +77,59 @@ def recording_add(self, key, data, *args, **kwargs):
7477
assert all(pte.find(engine) >= 0 for engine in streamed.values())
7578
# The files are gone once the program is saved.
7679
assert not glob.glob(str(tmp_path / "**" / "*.engine"), recursive=True)
80+
81+
82+
@pytest.mark.unit
83+
def test_engines_move_into_the_program_without_a_second_copy(tmp_path, monkeypatch):
84+
monkeypatch.setattr(tempfile, "tempdir", str(tmp_path))
85+
left = []
86+
real_write = FileBackedData.write_to_file
87+
88+
def recording_write(self, outfile, **kwargs):
89+
offset = real_write(self, outfile, **kwargs)
90+
left.append(os.path.getsize(self._path))
91+
return offset
92+
93+
monkeypatch.setattr(FileBackedData, "write_to_file", recording_write)
94+
torch.manual_seed(0)
95+
x = torch.randn(4, 64, device="cuda")
96+
torch_tensorrt.compile_and_save(
97+
torch.export.export(_TwoEngines().eval().cuda(), (x,)),
98+
tmp_path / "model.pte",
99+
output_format="executorch",
100+
arg_inputs=[x],
101+
min_block_size=1,
102+
torch_executed_ops={torch.ops.aten.sin.default},
103+
)
104+
# Each engine file was emptied as it was written, so no engine was ever on disk twice.
105+
assert left == [0, 0]
106+
107+
108+
@pytest.mark.unit
109+
def test_an_etrecord_makes_the_save_copy_the_engines(tmp_path, monkeypatch):
110+
monkeypatch.setattr(tempfile, "tempdir", str(tmp_path))
111+
left = []
112+
real_write = FileBackedData.write_to_file
113+
114+
def recording_write(self, outfile, **kwargs):
115+
offset = real_write(self, outfile, **kwargs)
116+
left.append(os.path.getsize(self._path))
117+
return offset
118+
119+
monkeypatch.setattr(FileBackedData, "write_to_file", recording_write)
120+
torch.manual_seed(0)
121+
x = torch.randn(4, 64, device="cuda")
122+
pte_path = tmp_path / "model.pte"
123+
torch_tensorrt.compile_and_save(
124+
torch.export.export(_TwoEngines().eval().cuda(), (x,)),
125+
pte_path,
126+
output_format="executorch",
127+
arg_inputs=[x],
128+
min_block_size=1,
129+
torch_executed_ops={torch.ops.aten.sin.default},
130+
save_options={"generate_etrecord": True},
131+
)
132+
assert pte_path.stat().st_size > 0
133+
assert (tmp_path / "model_etrecord.bin").stat().st_size > 0
134+
# The ETRecord reads the engines after the program is written, so they stayed in their files.
135+
assert left and all(size > 0 for size in left)

0 commit comments

Comments
 (0)