@@ -189,4 +189,124 @@ change drops it.
189189Use ``dynamically_allocate_resources=False `` to force static allocation inside the
190190context (the opposite direction — useful for profiling or benchmarking).
191191
192+ ----
193+
194+ Compiling from Memory-Mapped Weights
195+ ------------------------------------
196+
197+ .. note ::
198+
199+ ``map_weights ``, ``save_weights_for_mapping ``, ``checkpoint_source `` and ``low_memory `` are
200+ **experimental ** in 2.15: their API will change in 2.16.
201+
202+ To compile, a model's weights normally have to be loaded into host memory first, all of them, before
203+ the first engine is built. ``torch_tensorrt.dynamo.tools.map_weights `` instead points the model's weights at
204+ zero-copy views of a safetensors file:
205+
206+ - ``torch.export `` reads only shapes, so a weight's pages become resident only when TensorRT reads them,
207+ while it builds the engine that uses the weight.
208+ - The pages are dropped again once the last engine that reads the weight is built. A mapped weight stays
209+ valid; its pages are read back if it is used again.
210+
211+ ``torch_tensorrt.dynamo.low_memory `` does all of it in one block: it builds the model on ``meta ``, so
212+ building it allocates no weights, maps them, and drops their pages when the block ends, unless
213+ ``release_weights_after_build=False ``. Its ``compile_settings `` turn the resource partitioner on, so the
214+ model is built as several engines, each small enough to build within ``cpu_memory_budget `` (or the
215+ available memory without one), so only about one engine's weights are resident at a time. They also give a
216+ part of the model that runs in PyTorch its weights on the GPU after compilation
217+ (``offload_module_to_cpu ``). Pass them to ``compile ``:
218+
219+ .. code-block :: python
220+
221+ import torch
222+ import torch_tensorrt
223+
224+ # Map the checkpoint's weights, and build engines that fit an 8 GiB host memory budget.
225+ with torch_tensorrt.dynamo.low_memory(
226+ lambda : MyModel().eval(), " model.safetensors" , cpu_memory_budget = 8 << 30
227+ ) as low_mem:
228+ print (f " { low_mem.report.mapped_bytes / 2 ** 30 :.2f } GiB mapped " )
229+ example = torch.randn(1 , 3 , 224 , 224 ) # CPU: the mapped weights are CPU tensors
230+ exported = torch.export.export(low_mem.model, (example,))
231+ trt_model = torch_tensorrt.dynamo.compile(
232+ exported, inputs = [example.cuda()], ** low_mem.compile_settings
233+ )
234+
235+ ``cpu_memory_budget `` is the host memory the compiling process may use, counting what it already uses,
236+ and each engine gets up to a quarter of what's left. In :ref: `compile_from_mapped_weights `, a model with
237+ 192 MiB of weights built this way as 7 engines had at most 32 MiB of its weights file resident while
238+ compiling, against all 192 MiB when built as one engine.
239+
240+ ``low_memory `` takes the compile settings that lower memory during compilation:
241+ ``offload_module_to_cpu ``, ``enable_resource_partitioning ``, ``cpu_memory_budget `` and
242+ ``release_weights_after_build ``. ``compile_settings `` holds the ones given, with the first two on unless
243+ set. They apply only to the calls they're passed to: another model compiled inside the block keeps its own
244+ settings.
245+
246+ ``low_memory `` uses ``map_weights ``, which also works alone on a model you build yourself.
247+
248+ **What to know: **
249+
250+ - **Compile on the CPU. ** Mapped weights are CPU tensors, so keep the model and its example inputs on
251+ the CPU for ``torch.export ``. TensorRT takes weights as host pointers and reads each one in place:
252+ there's no GPU copy and no host copy of the weights.
253+ - **Checkpoints. ** Any safetensors file maps if its tensors have the names, dtypes and shapes the model
254+ uses; a mismatch raises. A checkpoint stored in other dtypes than the model runs in, for example
255+ float32 for a bfloat16 model, is converted once, as below. The file is checked from its header before
256+ anything is mapped: a truncated file, bytes after the last tensor, or tensors whose bytes overlap, leave
257+ gaps or don't hold their dtype and shape raise. Damage inside a tensor's bytes isn't detected, since
258+ that would mean reading the whole file. Tied weights, and distinct tensors that are one view of a
259+ storage (``w `` and ``Parameter(w.detach()) ``), are stored once and mapped to one view, so they stay
260+ shared; tensors that share a storage otherwise raise.
261+ - **Converting without loading. ** ``save_weights_for_mapping(model, path, source=...) `` writes a model
262+ built on ``meta `` straight from a checkpoint. It writes one tensor at a time. For each tensor it calls
263+ ``source(name, tensor) ``, which returns the data whole or in chunks along dimension 0, and it casts
264+ each chunk to the model's dtype. ``checkpoint_source(checkpoint, key=...) `` is a ready source that
265+ reads a safetensors file, or a sharded checkpoint's ``model.safetensors.index.json ``, in 64 MiB chunks,
266+ so even a float32 checkpoint is never loaded. Change the meta model the way the compiled model will be
267+ changed first, such as casting it to bfloat16, so the file holds the dtypes and shapes the export maps.
268+ For pi0.5, converting the 14.5 GB float32 checkpoint held 0.9 GiB of process memory, against 9.3 GiB
269+ for loading it on the CPU and writing the loaded model. A tensor the source doesn't store (it raises
270+ ``KeyError ``) is written from the model, with a warning if it is persistent, so buffers a checkpoint
271+ doesn't store, such as non-persistent rotary frequencies, must be real tensors, not on ``meta ``.
272+ accelerate's ``init_empty_weights() `` builds a model that way, and its persistent buffers, such as batch
273+ norm statistics, still come from the checkpoint. ``save_weights_for_mapping(model, path) `` without a
274+ source writes a model that is already loaded.
275+
276+ .. code-block :: python
277+
278+ from torch_tensorrt.dynamo.tools import checkpoint_source, save_weights_for_mapping
279+
280+ with torch.device(" meta" ):
281+ model = MyModel().to(torch.bfloat16)
282+ save_weights_for_mapping(
283+ model, " weights.safetensors" , source = checkpoint_source(" checkpoint.safetensors" )
284+ )
285+
286+ - **Missing names ** raise for any name in the model's ``state_dict ``, in ``map_weights `` and so in
287+ ``low_memory ``; non-persistent buffers, such as rotary frequencies, aren't required, so recompute those
288+ or write them with ``save_weights_for_mapping ``. With ``strict=False ``, ``map_weights `` lists missing
289+ names in its result's ``missing `` and leaves them as they were. A missing tensor on ``meta `` stays
290+ there, with no values, and a warning names it: a model that reads it can't be exported or compiled,
291+ and ``compile `` names it; one that never reads it is fine. On any error the model is left unchanged.
292+ - **The file must stay in place and unchanged ** while the model or its weights are in use: rewriting it
293+ changes or invalidates the weights. ``map_weights `` maps it copy-on-write and never writes to it. It
294+ reserves no memory for the mapping, so a file larger than RAM and swap maps too, except with
295+ ``vm.overcommit_memory=2 ``. Weights are read when compilation first uses them, so ``compile `` checks
296+ that the file of each mapped weight the program reads is still the one mapped (same inode, size and
297+ modification time) and raises otherwise; reading a weight past the end of a file truncated since would
298+ kill the process with ``SIGBUS ``. ``release_weights_after_build=True `` also frees weights that aren't
299+ mapped, which the source model then can't use.
300+ - **Releasing pages uses ** ``madvise ``. Where the process owns the weights file or may write to it,
301+ ``MADV_PAGEOUT `` (Linux 5.4 and later) drops a weight's pages and keeps writes made to it; it leaves
302+ pages another process maps too. Elsewhere, such as a root-owned file in a container image read by a
303+ normal user, ``MADV_DONTNEED `` drops them and discards such writes, so the weight reverts to the file's
304+ data. The bytes reported released are the pages that left the process's memory. On Windows the mapping
305+ works, but pages stay resident until the file is unmapped, and a warning says so.
306+
307+ With mapped weights, CPU compilation and the resource partitioner, the host's peak stays near the
308+ largest single engine build instead of the whole model. For pi0.5 (6.7 GiB of bfloat16 weights), host
309+ RSS before compilation went from 8.7 to 2.9 GiB. See :ref: `compile_from_mapped_weights ` for a runnable
310+ example.
311+
192312
0 commit comments