Skip to content

Commit fff69e3

Browse files
authored
Drop the "newer than 2.15" marks from the TensorRT backend page (#24214) (#24265)
Cherry-pick of #24214 onto release/1.6. The 1.6 release pins Torch-TensorRT 2.15, so the TensorRT backend page should not mark features as needing a version newer than 2.15. Docs only. The bot could not do this pick because an older pick branch for #24214 was left over, so this is the same commit applied by hand. The diff is identical to the commit on main, and the page now matches main.
1 parent d2e49e0 commit fff69e3

1 file changed

Lines changed: 14 additions & 17 deletions

File tree

‎docs/source/backends/tensorrt/tensorrt-overview.md‎

Lines changed: 14 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -3,22 +3,20 @@
33
The TensorRT backend runs ExecuTorch models on NVIDIA GPUs with
44
[TensorRT](https://developer.nvidia.com/tensorrt), NVIDIA's inference library. At export time,
55
TensorRT compiles the parts of the model it supports into engines (programs optimized for your
6-
GPU), and the engines are stored in the `.pte` file. At run time, the TensorRT delegate hands each
7-
engine back to TensorRT.
6+
GPU), and by default the engines are stored in the `.pte` file. At run time, the TensorRT delegate
7+
hands each engine back to TensorRT.
88

99
The delegate is built and shipped by the [Torch-TensorRT](https://github.com/pytorch/TensorRT)
1010
project, not by ExecuTorch. You export with Torch-TensorRT, and you run with the normal ExecuTorch
1111
runtime and its usual APIs. The
1212
[Torch-TensorRT ExecuTorch guide](https://docs.pytorch.org/TensorRT/user_guide/runtime_performance/executorch.html)
1313
covers every option in detail.
1414

15-
This page describes Torch-TensorRT 2.15 and newer. Features marked "newer than 2.15" need a later
16-
Torch-TensorRT build, such as a
17-
[nightly build](https://docs.pytorch.org/TensorRT/user_guide/runtime_performance/executorch.html#nightly-builds).
15+
This page describes Torch-TensorRT 2.15 and newer.
1816

1917
## Features
2018

21-
- **TensorRT engines inside a `.pte`.** One file holds the program and its engines.
19+
- **TensorRT engines inside a `.pte`.** By default, one file holds the program and its engines.
2220
- **Works with the CUDA backend in one program.** Operators that TensorRT cannot compile can go to
2321
the ExecuTorch [CUDA backend](../cuda/cuda-overview.md) instead of the CPU, so the whole model
2422
stays on the GPU.
@@ -36,8 +34,9 @@ Torch-TensorRT build, such as a
3634
- **Weight streaming.** An engine can keep part of its weights in CPU memory.
3735
- **Zero-copy KV cache.** The engine of a large language model (LLM) can update its key-value (KV)
3836
cache in place.
39-
- **Newer than 2.15:** engines in a separate data file, and optimization profiles (tunings for
40-
different input sizes) chosen per call.
37+
- **Engines in a separate data file.** The engines can be kept in a `.ptd` file next to the `.pte`.
38+
- **Optimization profiles.** One engine can carry tunings for different input sizes, and a C++
39+
application picks one per call.
4140

4241
## Target Requirements
4342

@@ -429,15 +428,15 @@ shows how to set the options that apply at run time, from C++ or from Python.
429428
Set how many bytes stay on the GPU with `weight_streaming_budget_per_engine` in
430429
`torch_tensorrt.save`, or with the `weight_streaming_budget` load option. The load option takes
431430
the number of bytes as a decimal string, or as an integer below 2 GiB.
432-
- [**Engines in a separate data file**](https://docs.pytorch.org/TensorRT/user_guide/runtime_performance/executorch.html#keep-the-engines-in-a-separate-file)
433-
(newer than 2.15). Pass `external_engine_data="engines"` to `torch_tensorrt.save` to write the
434-
engines to `engines.ptd` instead of the `.pte`, and pass that file when you load the program.
431+
- [**Engines in a separate data file.**](https://docs.pytorch.org/TensorRT/user_guide/runtime_performance/executorch.html#keep-the-engines-in-a-separate-file)
432+
Pass `external_engine_data="engines"` to `torch_tensorrt.save` to write the engines to
433+
`engines.ptd` instead of the `.pte`, and pass that file when you load the program.
435434
The Python loader takes one data file, so for a program that also uses the CUDA backend, pass
436435
`external_engine_data="aoti_cuda_blob"` to share its file.
437-
- [**Optimization profiles**](https://docs.pytorch.org/TensorRT/user_guide/runtime_performance/executorch.html#optimization-profiles)
438-
(newer than 2.15). One engine can be tuned for several ranges of input shapes, for example one
439-
for decoding one token at a time and one for a long prompt. A C++ application picks the profile for
440-
each call with `torch_tensorrt::executorch_backend::OptimizationProfileGuard`, from the
436+
- [**Optimization profiles.**](https://docs.pytorch.org/TensorRT/user_guide/runtime_performance/executorch.html#optimization-profiles)
437+
One engine can be tuned for several ranges of input shapes, for example one for decoding one
438+
token at a time and one for a long prompt. A C++ application picks the profile for each call
439+
with `torch_tensorrt::executorch_backend::OptimizationProfileGuard`, from the
441440
`torch_tensorrt/executorch/TensorRTBackend.h` header. Without the guard, every engine runs
442441
profile 0, and Python cannot choose a profile. The Python packages do not carry that header, and
443442
the guide explains where to get it.
@@ -450,6 +449,4 @@ shows how to set the options that apply at run time, from C++ or from Python.
450449
- [Export a model that keeps its data on the GPU](https://github.com/pytorch/TensorRT/blob/main/examples/torchtrt_executorch_example/export_device_resident.py)
451450
- [Export a decoder step whose KV cache stays in memory you own](https://github.com/pytorch/TensorRT/blob/main/examples/torchtrt_executorch_example/export_kv_cache_decode.py)
452451
- [Export one engine with two optimization profiles](https://github.com/pytorch/TensorRT/blob/main/examples/torchtrt_executorch_example/export_multi_profile.py)
453-
(newer than 2.15)
454452
- [C++ and Python reference runners](https://github.com/pytorch/TensorRT/tree/main/examples/executorch_reference_runner)
455-
(the two multi-profile programs need a build newer than 2.15)

0 commit comments

Comments
 (0)