You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit fff69e3
Browse filesBrowse the repository at this point in the historyBrowse files
Drop the "newer than 2.15" marks from the TensorRT backend page (#24214) (#24265)
Cherry-pick of #24214 onto release/1.6. The 1.6 release pins
Torch-TensorRT 2.15, so the TensorRT backend page should not mark
features as needing a version newer than 2.15. Docs only.
The bot could not do this pick because an older pick branch for #24214
was left over, so this is the same commit applied by hand. The diff is
identical to the commit on main, and the page now matches main.
This page describes Torch-TensorRT 2.15 and newer.
18
16
19
17
## Features
20
18
21
-
-**TensorRT engines inside a `.pte`.**One file holds the program and its engines.
19
+
-**TensorRT engines inside a `.pte`.**By default, one file holds the program and its engines.
22
20
-**Works with the CUDA backend in one program.** Operators that TensorRT cannot compile can go to
23
21
the ExecuTorch [CUDA backend](../cuda/cuda-overview.md) instead of the CPU, so the whole model
24
22
stays on the GPU.
@@ -36,8 +34,9 @@ Torch-TensorRT build, such as a
36
34
-**Weight streaming.** An engine can keep part of its weights in CPU memory.
37
35
-**Zero-copy KV cache.** The engine of a large language model (LLM) can update its key-value (KV)
38
36
cache in place.
39
-
-**Newer than 2.15:** engines in a separate data file, and optimization profiles (tunings for
40
-
different input sizes) chosen per call.
37
+
-**Engines in a separate data file.** The engines can be kept in a `.ptd` file next to the `.pte`.
38
+
-**Optimization profiles.** One engine can carry tunings for different input sizes, and a C++
39
+
application picks one per call.
41
40
42
41
## Target Requirements
43
42
@@ -429,15 +428,15 @@ shows how to set the options that apply at run time, from C++ or from Python.
429
428
Set how many bytes stay on the GPU with `weight_streaming_budget_per_engine` in
430
429
`torch_tensorrt.save`, or with the `weight_streaming_budget` load option. The load option takes
431
430
the number of bytes as a decimal string, or as an integer below 2 GiB.
432
-
-[**Engines in a separate data file**](https://docs.pytorch.org/TensorRT/user_guide/runtime_performance/executorch.html#keep-the-engines-in-a-separate-file)
433
-
(newer than 2.15). Pass `external_engine_data="engines"` to `torch_tensorrt.save` to write the
434
-
engines to `engines.ptd` instead of the `.pte`, and pass that file when you load the program.
431
+
-[**Engines in a separate data file.**](https://docs.pytorch.org/TensorRT/user_guide/runtime_performance/executorch.html#keep-the-engines-in-a-separate-file)
432
+
Pass `external_engine_data="engines"` to `torch_tensorrt.save` to write the engines to
433
+
`engines.ptd` instead of the `.pte`, and pass that file when you load the program.
435
434
The Python loader takes one data file, so for a program that also uses the CUDA backend, pass
436
435
`external_engine_data="aoti_cuda_blob"` to share its file.
One engine can be tuned for several ranges of input shapes, for example one for decoding one
438
+
token at a time and one for a long prompt. A C++ application picks the profile for each call
439
+
with `torch_tensorrt::executorch_backend::OptimizationProfileGuard`, from the
441
440
`torch_tensorrt/executorch/TensorRTBackend.h` header. Without the guard, every engine runs
442
441
profile 0, and Python cannot choose a profile. The Python packages do not carry that header, and
443
442
the guide explains where to get it.
@@ -450,6 +449,4 @@ shows how to set the options that apply at run time, from C++ or from Python.
450
449
-[Export a model that keeps its data on the GPU](https://github.com/pytorch/TensorRT/blob/main/examples/torchtrt_executorch_example/export_device_resident.py)
451
450
-[Export a decoder step whose KV cache stays in memory you own](https://github.com/pytorch/TensorRT/blob/main/examples/torchtrt_executorch_example/export_kv_cache_decode.py)
452
451
-[Export one engine with two optimization profiles](https://github.com/pytorch/TensorRT/blob/main/examples/torchtrt_executorch_example/export_multi_profile.py)
453
-
(newer than 2.15)
454
452
-[C++ and Python reference runners](https://github.com/pytorch/TensorRT/tree/main/examples/executorch_reference_runner)
455
-
(the two multi-profile programs need a build newer than 2.15)
0 commit comments