Skip to content

Commit ec35584

Browse files
cehongwanglanluo-nvidia
authored andcommitted
feat(dynamo): map_weights and low_memory - compile from memory-mapped weights, and convert a checkpoint without loading it (#4854)
1 parent 2c3e6e1 commit ec35584

17 files changed

Lines changed: 2246 additions & 39 deletions

File tree

‎docsrc/py_api/dynamo.rst‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,4 +32,29 @@ Functions
3232
Classes
3333
--------
3434

35+
.. autoclass:: low_memory
36+
37+
.. note::
38+
39+
``low_memory`` is **experimental** in 2.15: its API will change in 2.16.
40+
3541
.. autoclass:: CompilationSettings
42+
43+
Tools
44+
-----
45+
46+
Weight mapping: compile from a safetensors checkpoint's weights in place (see the resource management guide).
47+
48+
.. note::
49+
50+
These tools are **experimental** in 2.15: their API will change in 2.16.
51+
52+
.. currentmodule:: torch_tensorrt.dynamo.tools
53+
54+
.. autofunction:: map_weights
55+
56+
.. autofunction:: save_weights_for_mapping
57+
58+
.. autofunction:: checkpoint_source
59+
60+
.. autoclass:: MappedWeights

‎docsrc/tutorials/resource_memory/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,3 +14,4 @@ dynamic memory allocation, and low-CPU-memory compilation.
1414
Example: Weight Streaming <../_rendered_examples/dynamo/weight_streaming_example>
1515
Example: Dynamic Memory Allocation <../_rendered_examples/dynamo/dynamic_memory_allocation>
1616
Example: Low CPU Memory Compilation <../_rendered_examples/dynamo/low_cpu_memory_compilation>
17+
Example: Compiling from Memory-Mapped Weights <../_rendered_examples/dynamo/compile_from_mapped_weights>

‎docsrc/tutorials/resource_memory/resource_management.rst‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,4 +189,124 @@ change drops it.
189189
Use ``dynamically_allocate_resources=False`` to force static allocation inside the
190190
context (the opposite direction — useful for profiling or benchmarking).
191191

192+
----
193+
194+
Compiling from Memory-Mapped Weights
195+
------------------------------------
196+
197+
.. note::
198+
199+
``map_weights``, ``save_weights_for_mapping``, ``checkpoint_source`` and ``low_memory`` are
200+
**experimental** in 2.15: their API will change in 2.16.
201+
202+
To compile, a model's weights normally have to be loaded into host memory first, all of them, before
203+
the first engine is built. ``torch_tensorrt.dynamo.tools.map_weights`` instead points the model's weights at
204+
zero-copy views of a safetensors file:
205+
206+
- ``torch.export`` reads only shapes, so a weight's pages become resident only when TensorRT reads them,
207+
while it builds the engine that uses the weight.
208+
- The pages are dropped again once the last engine that reads the weight is built. A mapped weight stays
209+
valid; its pages are read back if it is used again.
210+
211+
``torch_tensorrt.dynamo.low_memory`` does all of it in one block: it builds the model on ``meta``, so
212+
building it allocates no weights, maps them, and drops their pages when the block ends, unless
213+
``release_weights_after_build=False``. Its ``compile_settings`` turn the resource partitioner on, so the
214+
model is built as several engines, each small enough to build within ``cpu_memory_budget`` (or the
215+
available memory without one), so only about one engine's weights are resident at a time. They also give a
216+
part of the model that runs in PyTorch its weights on the GPU after compilation
217+
(``offload_module_to_cpu``). Pass them to ``compile``:
218+
219+
.. code-block:: python
220+
221+
import torch
222+
import torch_tensorrt
223+
224+
# Map the checkpoint's weights, and build engines that fit an 8 GiB host memory budget.
225+
with torch_tensorrt.dynamo.low_memory(
226+
lambda: MyModel().eval(), "model.safetensors", cpu_memory_budget=8 << 30
227+
) as low_mem:
228+
print(f"{low_mem.report.mapped_bytes / 2**30:.2f} GiB mapped")
229+
example = torch.randn(1, 3, 224, 224) # CPU: the mapped weights are CPU tensors
230+
exported = torch.export.export(low_mem.model, (example,))
231+
trt_model = torch_tensorrt.dynamo.compile(
232+
exported, inputs=[example.cuda()], **low_mem.compile_settings
233+
)
234+
235+
``cpu_memory_budget`` is the host memory the compiling process may use, counting what it already uses,
236+
and each engine gets up to a quarter of what's left. In :ref:`compile_from_mapped_weights`, a model with
237+
192 MiB of weights built this way as 7 engines had at most 32 MiB of its weights file resident while
238+
compiling, against all 192 MiB when built as one engine.
239+
240+
``low_memory`` takes the compile settings that lower memory during compilation:
241+
``offload_module_to_cpu``, ``enable_resource_partitioning``, ``cpu_memory_budget`` and
242+
``release_weights_after_build``. ``compile_settings`` holds the ones given, with the first two on unless
243+
set. They apply only to the calls they're passed to: another model compiled inside the block keeps its own
244+
settings.
245+
246+
``low_memory`` uses ``map_weights``, which also works alone on a model you build yourself.
247+
248+
**What to know:**
249+
250+
- **Compile on the CPU.** Mapped weights are CPU tensors, so keep the model and its example inputs on
251+
the CPU for ``torch.export``. TensorRT takes weights as host pointers and reads each one in place:
252+
there's no GPU copy and no host copy of the weights.
253+
- **Checkpoints.** Any safetensors file maps if its tensors have the names, dtypes and shapes the model
254+
uses; a mismatch raises. A checkpoint stored in other dtypes than the model runs in, for example
255+
float32 for a bfloat16 model, is converted once, as below. The file is checked from its header before
256+
anything is mapped: a truncated file, bytes after the last tensor, or tensors whose bytes overlap, leave
257+
gaps or don't hold their dtype and shape raise. Damage inside a tensor's bytes isn't detected, since
258+
that would mean reading the whole file. Tied weights, and distinct tensors that are one view of a
259+
storage (``w`` and ``Parameter(w.detach())``), are stored once and mapped to one view, so they stay
260+
shared; tensors that share a storage otherwise raise.
261+
- **Converting without loading.** ``save_weights_for_mapping(model, path, source=...)`` writes a model
262+
built on ``meta`` straight from a checkpoint. It writes one tensor at a time. For each tensor it calls
263+
``source(name, tensor)``, which returns the data whole or in chunks along dimension 0, and it casts
264+
each chunk to the model's dtype. ``checkpoint_source(checkpoint, key=...)`` is a ready source that
265+
reads a safetensors file, or a sharded checkpoint's ``model.safetensors.index.json``, in 64 MiB chunks,
266+
so even a float32 checkpoint is never loaded. Change the meta model the way the compiled model will be
267+
changed first, such as casting it to bfloat16, so the file holds the dtypes and shapes the export maps.
268+
For pi0.5, converting the 14.5 GB float32 checkpoint held 0.9 GiB of process memory, against 9.3 GiB
269+
for loading it on the CPU and writing the loaded model. A tensor the source doesn't store (it raises
270+
``KeyError``) is written from the model, with a warning if it is persistent, so buffers a checkpoint
271+
doesn't store, such as non-persistent rotary frequencies, must be real tensors, not on ``meta``.
272+
accelerate's ``init_empty_weights()`` builds a model that way, and its persistent buffers, such as batch
273+
norm statistics, still come from the checkpoint. ``save_weights_for_mapping(model, path)`` without a
274+
source writes a model that is already loaded.
275+
276+
.. code-block:: python
277+
278+
from torch_tensorrt.dynamo.tools import checkpoint_source, save_weights_for_mapping
279+
280+
with torch.device("meta"):
281+
model = MyModel().to(torch.bfloat16)
282+
save_weights_for_mapping(
283+
model, "weights.safetensors", source=checkpoint_source("checkpoint.safetensors")
284+
)
285+
286+
- **Missing names** raise for any name in the model's ``state_dict``, in ``map_weights`` and so in
287+
``low_memory``; non-persistent buffers, such as rotary frequencies, aren't required, so recompute those
288+
or write them with ``save_weights_for_mapping``. With ``strict=False``, ``map_weights`` lists missing
289+
names in its result's ``missing`` and leaves them as they were. A missing tensor on ``meta`` stays
290+
there, with no values, and a warning names it: a model that reads it can't be exported or compiled,
291+
and ``compile`` names it; one that never reads it is fine. On any error the model is left unchanged.
292+
- **The file must stay in place and unchanged** while the model or its weights are in use: rewriting it
293+
changes or invalidates the weights. ``map_weights`` maps it copy-on-write and never writes to it. It
294+
reserves no memory for the mapping, so a file larger than RAM and swap maps too, except with
295+
``vm.overcommit_memory=2``. Weights are read when compilation first uses them, so ``compile`` checks
296+
that the file of each mapped weight the program reads is still the one mapped (same inode, size and
297+
modification time) and raises otherwise; reading a weight past the end of a file truncated since would
298+
kill the process with ``SIGBUS``. ``release_weights_after_build=True`` also frees weights that aren't
299+
mapped, which the source model then can't use.
300+
- **Releasing pages uses** ``madvise``. Where the process owns the weights file or may write to it,
301+
``MADV_PAGEOUT`` (Linux 5.4 and later) drops a weight's pages and keeps writes made to it; it leaves
302+
pages another process maps too. Elsewhere, such as a root-owned file in a container image read by a
303+
normal user, ``MADV_DONTNEED`` drops them and discards such writes, so the weight reverts to the file's
304+
data. The bytes reported released are the pages that left the process's memory. On Windows the mapping
305+
works, but pages stay resident until the file is unmapped, and a warning says so.
306+
307+
With mapped weights, CPU compilation and the resource partitioner, the host's peak stays near the
308+
largest single engine build instead of the whole model. For pi0.5 (6.7 GiB of bfloat16 weights), host
309+
RSS before compilation went from 8.7 to 2.9 GiB. See :ref:`compile_from_mapped_weights` for a runnable
310+
example.
311+
192312

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: BSD-3-Clause
3+
4+
"""
5+
6+
.. _compile_from_mapped_weights:
7+
8+
Compiling from Memory-Mapped Weights
9+
====================================
10+
11+
To compile a model, its weights normally have to be loaded into host memory first, all of them, before
12+
the first TensorRT engine is built. ``torch_tensorrt.dynamo.tools.map_weights`` instead points the model's
13+
weights at zero-copy views of a safetensors file. ``torch.export`` reads only shapes, so a weight's
14+
pages become resident only when TensorRT reads them, while it builds the engine that uses the weight.
15+
The pages are dropped again once the last engine that reads the weight is built. With the resource
16+
partitioner, the model is built as several engines, each small enough to build within a host memory
17+
budget, so only about one engine's weights are resident at a time.
18+
19+
This example:
20+
21+
1. writes a model's weights once with ``save_weights_for_mapping``, standing in for a checkpoint;
22+
2. with ``torch_tensorrt.dynamo.low_memory``, builds the model on the ``meta`` device, so constructing it
23+
allocates no weights, maps its weights, and exports and compiles it on the CPU as several engines
24+
within a CPU memory budget, watching how much of the weights file is resident;
25+
3. checks the compiled model against the original.
26+
27+
How much of the weights file is resident is read from ``/proc/self/smaps``, so it prints on Linux only.
28+
"""
29+
30+
import os
31+
import tempfile
32+
import threading
33+
from typing import Any, Callable, Tuple
34+
35+
import psutil
36+
import torch
37+
import torch_tensorrt
38+
from torch_tensorrt.dynamo.tools import save_weights_for_mapping
39+
40+
41+
def resident_mib(path: str) -> float:
42+
"""MiB of the file at `path` resident in this process (its mappings' Rss in /proc/self/smaps)."""
43+
if not os.path.exists("/proc/self/smaps"):
44+
return float("nan")
45+
total, in_file = 0, False
46+
for line in open("/proc/self/smaps"):
47+
fields = line.split()
48+
if fields and "-" in fields[0] and len(fields) >= 5: # a mapping's header line
49+
in_file = fields[-1] == path
50+
elif in_file and fields[0] == "Rss:":
51+
total += int(fields[1])
52+
return total / 1024
53+
54+
55+
def peak_resident_mib(path: str, fn: Callable[[], Any]) -> Tuple[Any, float]:
56+
"""Run ``fn()`` while sampling ``resident_mib(path)``; return its result and the peak. TensorRT
57+
releases the GIL while it builds an engine, so the samples cover the builds."""
58+
peak, done = [resident_mib(path)], threading.Event()
59+
60+
def sample() -> None:
61+
while not done.is_set():
62+
peak[0] = max(peak[0], resident_mib(path))
63+
done.wait(0.005)
64+
65+
sampler = threading.Thread(target=sample)
66+
sampler.start()
67+
try:
68+
result = fn()
69+
finally:
70+
done.set()
71+
sampler.join()
72+
return result, peak[0]
73+
74+
75+
class MLP(torch.nn.Module):
76+
def __init__(self, width: int = 2048, depth: int = 12) -> None:
77+
super().__init__()
78+
self.layers = torch.nn.ModuleList(
79+
torch.nn.Linear(width, width) for _ in range(depth)
80+
)
81+
82+
def forward(self, x: torch.Tensor) -> torch.Tensor:
83+
for layer in self.layers:
84+
x = torch.relu(layer(x))
85+
return x
86+
87+
88+
# %%
89+
# Write the weights once
90+
# ----------------------
91+
#
92+
# Any safetensors file maps if its tensors have the names, dtypes and shapes the model uses.
93+
# ``save_weights_for_mapping`` is for checkpoints whose loader casts, e.g. stored in float32 and run in
94+
# bfloat16: it writes the weights as they are after loading, a tied weight once.
95+
96+
torch.manual_seed(0)
97+
reference = MLP().eval()
98+
path = os.path.join(tempfile.mkdtemp(), "mlp.safetensors")
99+
save_weights_for_mapping(reference, path)
100+
print(f"wrote {os.path.getsize(path) / 2**20:.0f} MiB of weights to {path}")
101+
102+
# %%
103+
# Build, map, export and compile within a memory budget
104+
# -----------------------------------------------------
105+
#
106+
# ``low_memory`` builds the model on ``meta``, where it allocates nothing, and maps its weights; every
107+
# tensor must be in the file. Mapping doesn't read the weights, so none of the file is resident yet.
108+
#
109+
# The mapped weights are CPU tensors, so the model and its example input stay on the CPU through
110+
# ``torch.export``. TensorRT takes weights as host pointers and reads each one in place from the mapping:
111+
# there's no GPU copy and no host copy. A weight's pages are dropped once the last engine that reads it
112+
# is built; the tensor stays valid and is re-read from the file if used.
113+
#
114+
# ``enable_resource_partitioning`` splits the model into engines small enough to build within
115+
# ``cpu_memory_budget``, the host memory the process may use. Each engine gets up to a quarter of the
116+
# budget left over after the memory already in use, since building an engine takes a few times its
117+
# weights' size. So a budget of the memory in use now plus the weights' size makes engines of at most
118+
# about a quarter of the weights, and building them one after another keeps only about one engine's
119+
# weights resident.
120+
121+
x = torch.randn(4, 2048)
122+
with (
123+
torch_tensorrt.dynamo.low_memory(
124+
lambda: MLP().eval(),
125+
path,
126+
cpu_memory_budget=psutil.Process().memory_info().rss + os.path.getsize(path),
127+
) as low_mem,
128+
torch.no_grad(),
129+
):
130+
print(
131+
f"mapped {low_mem.report.mapped_bytes / 2**20:.0f} MiB; "
132+
f"resident: {resident_mib(path):.0f} MiB"
133+
)
134+
exported = torch.export.export(low_mem.model, (x,))
135+
trt_model, peak = peak_resident_mib(
136+
path,
137+
lambda: torch_tensorrt.dynamo.compile(
138+
exported, inputs=[x.cuda()], **low_mem.compile_settings
139+
),
140+
)
141+
engines = [name for name, _ in trt_model.named_children() if "_run_on_acc" in name]
142+
# As one engine, without the partitioner, the whole file (192 MiB here) is resident while it builds.
143+
# With release_weights_after_build=False, all of it would stay resident after compiling.
144+
print(
145+
f"compiled into {len(engines)} engines; at most {peak:.0f} MiB of the weights file was "
146+
f"resident, {resident_mib(path):.0f} MiB is now"
147+
)
148+
149+
# %%
150+
# Check the result
151+
# ----------------
152+
153+
with torch.no_grad():
154+
expected = reference(x)
155+
actual = trt_model(x.cuda()).cpu()
156+
torch.testing.assert_close(actual, expected, rtol=1e-2, atol=1e-2)
157+
print(
158+
f"max abs difference from the original model: {(actual - expected).abs().max().item():.2e}"
159+
)
160+
161+
# %%
162+
# The file must stay in place while the model or its weights are in use: released pages are read from it
163+
# again. On large models this keeps the host's peak near the largest single engine build instead of the
164+
# whole model, e.g. pi0.5 (6.7 GiB of bfloat16 weights): host RSS before compilation went from 8.7 to
165+
# 2.9 GiB.

‎py/torch_tensorrt/dynamo/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
save_cross_compiled_exported_program,
1919
)
2020
from ._exporter import export
21+
from ._low_memory import low_memory
2122
from ._refit import refit_module_weights
2223
from ._settings import CompilationSettings
2324
from ._SourceIR import SourceIR

0 commit comments

Comments
 (0)