Skip to content

Commit 3cf1702

Browse files
committed
fix(executorch): move compile_and_save's engines into the program instead of copying them
`compile_and_save` writes each engine to a temporary file as soon as it is built. The ExecuTorch save then copies that file into the `.pte` (or the `.ptd`), and the file is deleted only when the call returns. So while the program is written, every engine is on disk twice. For a model whose engine is tens of gigabytes, the copy needs that much more free disk, and an export can fail on a device that has room for the result. The backend now hands each of these engine files to ExecuTorch as data that owns the file (`FileBackedData(path, cleanup=True)`); it is deleted after the save anyway. Inside `compile_and_save`, the save then writes the program with `consume=True`, so ExecuTorch moves each engine into the program: it copies it back to front and truncates the file as it goes, and each engine is on disk once. ExecuTorch does this only when the temporary directory, where the engine files are, is on the same file system as the program; otherwise it copies. The save checks that ExecuTorch's `write_to_file` takes `consume`; an ExecuTorch without it keeps copying, as before. After the move the engine file is empty, while the compiled module still maps it. Reading the mapping then is a bus error. The ETRecord reads the engines, so when the save consumes, the ETRecord is saved before the program, and an ETRecord that can't be saved stops the save before the program is written. Every other save keeps the old order: the program is written first, so it is kept even when the ETRecord can't be saved. Test plan: - `tests/py/dynamo/executorch/test_api.py`, CPU, with the lowering stubbed: - New `test_compile_and_save_saves_the_etrecord_before_the_engines_move` checks the calls inside `compile_and_save`'s offload block: the ETRecord first, then the program with `consume=True`. It fails without this change. The `.ptd` write consumes too. - New `test_compile_and_save_copies_with_an_executorch_that_cannot_consume`: with an ExecuTorch whose writes take no `consume`, the same save copies, in the old order. - New `test_save_executorch_keeps_the_program_when_the_etrecord_fails`: a plain save whose ETRecord save raises still writes the `.pte` first, without `consume`. - `tests/py/dynamo/executorch/test_compile_and_save.py` on an A100: - New `test_engines_move_into_the_program_without_a_second_copy` checks that each engine file is empty after the program is written. It skips, with a reason, when ExecuTorch can only copy. With an ExecuTorch that can move and without this change, the files keep their full size. - New `test_an_etrecord_is_saved_from_the_engines_before_they_move`. With the move but the old order, the process dies with a bus error.
1 parent 14a181c commit 3cf1702

5 files changed

Lines changed: 169 additions & 14 deletions

File tree

‎py/torch_tensorrt/_compile.py‎

Lines changed: 30 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@
7777
from torch_tensorrt.dynamo._tracer import trace as dynamo_trace
7878
from torch_tensorrt.dynamo.runtime import TorchTensorRTModule
7979
from torch_tensorrt.dynamo.runtime._offloaded_engines import (
80+
defer_engine_setup_to_load,
8081
offload_engines_to_disk,
8182
)
8283
from torch_tensorrt.dynamo.utils import get_torch_inputs
@@ -1550,6 +1551,9 @@ def compile_and_save(
15501551
but each engine is written to a file as soon as it is built, so no built engine is held on the GPU
15511552
or in host memory while the next ones build. With ``output_format="executorch"`` no engine is ever
15521553
loaded: the save streams each file into the ``.pte`` (or the ``.ptd`` with ``external_engine_data``).
1554+
When ExecuTorch can consume file-backed data on a write, and the temporary directory (``TMPDIR``) is
1555+
on the same file system as ``file_path``, the save moves each file there instead, so no engine is on
1556+
disk twice.
15531557
The other formats save each engine from a loaded one, so their engines are loaded once all of them
15541558
are built, and their save needs as much memory as after ``compile``.
15551559
@@ -1600,7 +1604,9 @@ def compile_and_save(
16001604
save(trt_gm, os.fspath(file_path), output_format=output_format, **save_kwargs)
16011605

16021606

1603-
def _write_external_tensor_data(executorch_program: Any, file_path: str) -> None:
1607+
def _write_external_tensor_data(
1608+
executorch_program: Any, file_path: str, consume: bool = False
1609+
) -> None:
16041610
"""Write an ExecuTorch program's external named tensor data (``.ptd``) next to the ``.pte``.
16051611
16061612
The CUDA (AOTInductor) backend emits its weights as external named data
@@ -1616,7 +1622,10 @@ def _write_external_tensor_data(executorch_program: Any, file_path: str) -> None
16161622
# Direct attribute access (no getattr default) so a future rename fails loud.
16171623
if executorch_program._tensor_data:
16181624
out_dir = os.path.dirname(os.path.abspath(file_path))
1619-
executorch_program.write_tensor_data_to_file(out_dir)
1625+
if consume:
1626+
executorch_program.write_tensor_data_to_file(out_dir, consume=True)
1627+
else:
1628+
executorch_program.write_tensor_data_to_file(out_dir)
16201629
# write_tensor_data_to_file adds .ptd unless the name already ends with it.
16211630
paths = [
16221631
os.path.join(out_dir, name if name.endswith(".ptd") else f"{name}.ptd")
@@ -1752,13 +1761,26 @@ def _lower_and_write_executorch(
17521761
# the same call is the caller's to make, because finalization happens on
17531762
# ExecuTorch's own manager and out of this library's sight.
17541763
check_zero_copy_kv(executorch_program)
1764+
# Follows ExecuTorch's example convention (e.g. examples/cuda/scripts/export.py):
1765+
# persist the ETRecord as "<pte_base>_etrecord.bin" next to the .pte.
1766+
etrecord_path = os.path.splitext(file_path)[0] + "_etrecord.bin"
1767+
# Inside compile_and_save the engines are in files deleted after this save, so the writes move
1768+
# them into the program instead of copying them, when ExecuTorch can. The ETRecord reads the
1769+
# engines, so then it is saved first. Elsewhere it is saved last, so the program is kept even
1770+
# when the ETRecord can't be saved.
1771+
consume = (
1772+
defer_engine_setup_to_load()
1773+
and "consume" in inspect.signature(executorch_program.write_to_file).parameters
1774+
)
1775+
if generate_etrecord and consume:
1776+
executorch_program.get_etrecord().save(etrecord_path)
17551777
with open(file_path, "wb") as f:
1756-
executorch_program.write_to_file(f)
1757-
_write_external_tensor_data(executorch_program, file_path)
1758-
if generate_etrecord:
1759-
# Follows ExecuTorch's example convention (e.g. examples/cuda/scripts/export.py):
1760-
# persist the ETRecord as "<pte_base>_etrecord.bin" next to the .pte.
1761-
etrecord_path = os.path.splitext(file_path)[0] + "_etrecord.bin"
1778+
if consume:
1779+
executorch_program.write_to_file(f, consume=True)
1780+
else:
1781+
executorch_program.write_to_file(f)
1782+
_write_external_tensor_data(executorch_program, file_path, consume)
1783+
if generate_etrecord and not consume:
17621784
executorch_program.get_etrecord().save(etrecord_path)
17631785

17641786

‎py/torch_tensorrt/dynamo/runtime/_offloaded_engines.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
engines built so far are then held neither on the GPU nor in host memory while the next ones build.
99
``compile_module`` leaves them in their files: an ExecuTorch save streams each file into the program, and
1010
``compile_and_save`` loads them before any other save. The files are deleted when the block ends.
11+
A file the save moved stays mapped, empty, until then, so nothing may read its engine after the save.
1112
"""
1213

1314
import contextlib

‎py/torch_tensorrt/executorch/backend.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -699,7 +699,9 @@ def preprocess(
699699

700700
engine_data: Any
701701
if engine_file is not None:
702-
engine_data = FileBackedData(engine_file)
702+
# compile_and_save deletes the file after the save, so the data can own it, which lets
703+
# a consuming write move the engine into the program instead of copying it.
704+
engine_data = FileBackedData(engine_file, cleanup=True)
703705
engine_data_sha256 = engine_data.sha256()
704706
else:
705707
assert isinstance(engine_bytes, bytes)

‎tests/py/dynamo/executorch/test_api.py‎

Lines changed: 88 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2824,39 +2824,63 @@ def test_lift_keeps_parameter_for_any_dtype(dtype, requires_grad):
28242824
# next to the .pte.
28252825

28262826

2827-
def _patch_executorch_lowering(monkeypatch, captured):
2827+
def _patch_executorch_lowering(
2828+
monkeypatch, captured, log=None, etrecord_error=None, can_consume=True
2829+
):
28282830
"""Spy the seams _save_as_executorch delegates to after #4440, without running a
28292831
real TRT lowering. Records the kwargs forwarded into
28302832
torch_tensorrt.executorch.export() (captured["export_kwargs"]); lets the real
28312833
export() run against an engine-free program while stubbing only the innermost
28322834
ExecuTorch lowering, so the kwargs export() finally hands that lowering --
28332835
including the get_edge_compile_config() it substitutes for an omitted
28342836
compile_config -- land in `captured`; and records backend_config from
2835-
to_executorch(). Fills `captured`; returns nothing."""
2837+
to_executorch(). Fills `captured`; returns nothing. With `log`, records the order of
2838+
the ETRecord, program and .ptd writes, and whether each write consumes;
2839+
`etrecord_error` makes the ETRecord save raise; `can_consume=False` stands in for
2840+
an ExecuTorch whose writes can only copy."""
28362841
import executorch.exir as exir
28372842
import torch_tensorrt._compile as tc
28382843
import torch_tensorrt.executorch as tte
28392844

28402845
class _FakeETRecord:
28412846
def save(self, path):
2847+
if log is not None:
2848+
log.append("etrecord")
2849+
if etrecord_error is not None:
2850+
raise etrecord_error
28422851
with open(path, "wb") as fh:
28432852
fh.write(b"etrecord")
28442853

28452854
class _FakeExec:
2846-
def write_to_file(self, f):
2855+
_tensor_data = {"model": b""}
2856+
2857+
def write_to_file(self, f, consume=False):
2858+
if log is not None:
2859+
log.append(("program", consume))
28472860
f.write(b"")
28482861

2862+
def write_tensor_data_to_file(self, outdir, consume=False):
2863+
if log is not None:
2864+
log.append(("ptd", consume))
2865+
28492866
def get_etrecord(self):
28502867
return _FakeETRecord()
28512868

2869+
class _CopyOnlyExec(_FakeExec):
2870+
def write_to_file(self, f):
2871+
super().write_to_file(f)
2872+
2873+
def write_tensor_data_to_file(self, outdir):
2874+
super().write_tensor_data_to_file(outdir)
2875+
28522876
class _FakeEdge:
28532877
# export() reorders each method's mutations after lowering, over every
28542878
# method the manager holds. No method here holds a program to reorder.
28552879
methods = ()
28562880

28572881
def to_executorch(self, config=None):
28582882
captured["backend_config"] = config
2859-
return _FakeExec()
2883+
return _FakeExec() if can_consume else _CopyOnlyExec()
28602884

28612885
# Innermost real lowering seam, called inside export(): capture the kwargs it
28622886
# receives, notably compile_config already resolved to its default when omitted.
@@ -2875,7 +2899,6 @@ def _spy_export(source, **kw):
28752899
return real_export(source, **kw)
28762900

28772901
monkeypatch.setattr(tte, "export", _spy_export)
2878-
monkeypatch.setattr(tc, "_write_external_tensor_data", lambda prog, path: None)
28792902
# ENABLED_FEATURES is an immutable namedtuple; swap the whole module attribute.
28802903
monkeypatch.setattr(
28812904
tc, "ENABLED_FEATURES", types.SimpleNamespace(torch_tensorrt_runtime=True)
@@ -2933,6 +2956,66 @@ def test_save_executorch_forwards_lowering_kwargs(monkeypatch, tmp_path):
29332956
assert (tmp_path / "model_etrecord.bin").exists()
29342957

29352958

2959+
@pytest.mark.unit
2960+
def test_save_executorch_keeps_the_program_when_the_etrecord_fails(
2961+
monkeypatch, tmp_path
2962+
):
2963+
pytest.importorskip("executorch.exir")
2964+
import torch_tensorrt._compile as tc
2965+
2966+
log = []
2967+
_patch_executorch_lowering(
2968+
monkeypatch, {}, log=log, etrecord_error=RuntimeError("cannot serialize")
2969+
)
2970+
out = tmp_path / "model.pte"
2971+
ep = torch.export.export(_AddOne(), (torch.randn(2, 2),))
2972+
with pytest.raises(RuntimeError, match="cannot serialize"):
2973+
tc._save_as_executorch(ep, str(out), generate_etrecord=True)
2974+
2975+
assert log == [("program", False), ("ptd", False), "etrecord"]
2976+
assert out.exists()
2977+
2978+
2979+
@pytest.mark.unit
2980+
def test_compile_and_save_saves_the_etrecord_before_the_engines_move(
2981+
monkeypatch, tmp_path
2982+
):
2983+
pytest.importorskip("executorch.exir")
2984+
import torch_tensorrt._compile as tc
2985+
from torch_tensorrt.dynamo.runtime._offloaded_engines import (
2986+
offload_engines_to_disk,
2987+
)
2988+
2989+
log = []
2990+
_patch_executorch_lowering(monkeypatch, {}, log=log)
2991+
ep = torch.export.export(_AddOne(), (torch.randn(2, 2),))
2992+
# The block compile_and_save runs its save in, where writing the program moves the engines out
2993+
# of the files the ETRecord reads.
2994+
with offload_engines_to_disk():
2995+
tc._save_as_executorch(ep, str(tmp_path / "model.pte"), generate_etrecord=True)
2996+
2997+
assert log == ["etrecord", ("program", True), ("ptd", True)]
2998+
2999+
3000+
@pytest.mark.unit
3001+
def test_compile_and_save_copies_with_an_executorch_that_cannot_consume(
3002+
monkeypatch, tmp_path
3003+
):
3004+
pytest.importorskip("executorch.exir")
3005+
import torch_tensorrt._compile as tc
3006+
from torch_tensorrt.dynamo.runtime._offloaded_engines import (
3007+
offload_engines_to_disk,
3008+
)
3009+
3010+
log = []
3011+
_patch_executorch_lowering(monkeypatch, {}, log=log, can_consume=False)
3012+
ep = torch.export.export(_AddOne(), (torch.randn(2, 2),))
3013+
with offload_engines_to_disk():
3014+
tc._save_as_executorch(ep, str(tmp_path / "model.pte"), generate_etrecord=True)
3015+
3016+
assert log == [("program", False), ("ptd", False), "etrecord"]
3017+
3018+
29363019
@pytest.mark.unit
29373020
def test_save_executorch_defaults_when_lowering_kwargs_omitted(monkeypatch, tmp_path):
29383021
pytest.importorskip("executorch.exir")

‎tests/py/dynamo/executorch/test_compile_and_save.py‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
# SPDX-License-Identifier: BSD-3-Clause
33

44
import glob
5+
import inspect
56
import os
67
import tempfile
78

@@ -74,3 +75,49 @@ def recording_add(self, key, data, *args, **kwargs):
7475
assert all(pte.find(engine) >= 0 for engine in streamed.values())
7576
# The files are gone once the program is saved.
7677
assert not glob.glob(str(tmp_path / "**" / "*.engine"), recursive=True)
78+
79+
80+
@pytest.mark.unit
81+
def test_engines_move_into_the_program_without_a_second_copy(tmp_path, monkeypatch):
82+
if "consume" not in inspect.signature(FileBackedData.write_to_file).parameters:
83+
pytest.skip("This ExecuTorch can only copy file-backed data")
84+
monkeypatch.setattr(tempfile, "tempdir", str(tmp_path))
85+
left = []
86+
real_write = FileBackedData.write_to_file
87+
88+
def recording_write(self, outfile, **kwargs):
89+
real_write(self, outfile, **kwargs)
90+
left.append(os.path.getsize(self._path))
91+
92+
monkeypatch.setattr(FileBackedData, "write_to_file", recording_write)
93+
torch.manual_seed(0)
94+
x = torch.randn(4, 64, device="cuda")
95+
torch_tensorrt.compile_and_save(
96+
torch.export.export(_TwoEngines().eval().cuda(), (x,)),
97+
tmp_path / "model.pte",
98+
output_format="executorch",
99+
arg_inputs=[x],
100+
min_block_size=1,
101+
torch_executed_ops={torch.ops.aten.sin.default},
102+
)
103+
# Each engine file was emptied as it was written, so no engine was ever on disk twice.
104+
assert left == [0, 0]
105+
106+
107+
@pytest.mark.unit
108+
def test_an_etrecord_is_saved_from_the_engines_before_they_move(tmp_path, monkeypatch):
109+
monkeypatch.setattr(tempfile, "tempdir", str(tmp_path))
110+
torch.manual_seed(0)
111+
x = torch.randn(4, 64, device="cuda")
112+
pte_path = tmp_path / "model.pte"
113+
torch_tensorrt.compile_and_save(
114+
torch.export.export(_TwoEngines().eval().cuda(), (x,)),
115+
pte_path,
116+
output_format="executorch",
117+
arg_inputs=[x],
118+
min_block_size=1,
119+
torch_executed_ops={torch.ops.aten.sin.default},
120+
save_options={"generate_etrecord": True},
121+
)
122+
assert pte_path.stat().st_size > 0
123+
assert (tmp_path / "model_etrecord.bin").stat().st_size > 0

0 commit comments

Comments
 (0)