Skip to content

Commit 0720451

Browse files
[PyBindings] Torch-free follow-ups: reused inputs, error reasons, bfloat16 shape, dtype names, tests, docs (#24209)
## 1. Stop reusing overwritten inputs **Problem.** Export lets later values reuse an input's planned memory, so a second run without new inputs gave wrong numbers silently: ```python method.set_inputs([x, y]); method.execute() # right method.execute() # wrong, silently ``` So did `Module.plan_execute()` and partial input lists. **Change.** `Method` installs memory-planned inputs again from what the caller passed at every `execute()`; `Module` does the same for kept inputs a call omits. So inputs are read at each run: in-place edits are seen, a storage change raises "pass it again", and an input the method writes in place (`state.add_(x)`) no longer carries over, unlike repeated `Method::execute()` in C++. The README notes the limits for DLPack producers and `clone_outputs=False` results passed back in. ## 2. Say why a call was refused The runtime logs why it refused a call but returns only a code, so exceptions said just `error 0x10` or `error: 0x22`. Now they include the error lines the runtime logged during the call: ``` RuntimeError: Failed to load method forward, error: 0x22: Failed to load /tmp/executorch_cuda_..._0.so with error: libgomp.so.1: cannot open shared object file ... ``` ## 3. Keep the shape of bfloat16 results A (2, 3) bfloat16 result showed up in the buffer protocol as 12 flat bytes. Now dtypes without a buffer format show as same-width integers in the real shape; DLPack still carries the real dtype. Pass a bfloat16 result back in as the result or its bytes, not `np.asarray(result)`. ## 4. Report a dtype by its name `ExecuTorchResult.dtype` now returns a string such as `"bfloat16"` instead of a NumPy dtype (which raised for bfloat16), and the new `TensorInfo.dtype_name()` returns the same name. The experimental `dtype_name` property is removed. `TensorInfo.dtype()` still returns the integer; its repr prints `dtype=float32`. ## 5. Docs The API reference keeps torch tensors in and out as the main example. A short "Without PyTorch" section shows the fallback: NumPy arrays in, `ExecuTorchResult` values out, and DLPack (for example CuPy) for GPU data. The `Method.execute` docstring follows the same order. ## Testing On Linux x86_64 with an NVIDIA GPU: - Two new tests in `test_pybindings.py` (inputs installed again at each run, and the reason in error messages), plus small edits to existing tests for the bfloat16 buffer view and dtype names. Both new tests, and most of the edited ones, fail on the base branch; all of them pass here. - The existing pybindings tests, including the no-ATen file as CI runs it and the CUDA DLPack tests in both output modes: no new failures. A few tests fail here because a library is missing (quantized ops, pandas); this change does not touch them. - Without PyTorch and libgomp, the CUDA load error now names the missing library. - The basic torch example and the "Without PyTorch" example on the API reference page print the documented output. The second one, and its CuPy steps, also ran with PyTorch imports blocked. - Not tested: the ATen build, macOS, and Windows. Co-authored-by: Anthony Shoumikhin <anthony@shoumikh.in>
1 parent f76d027 commit 0720451

9 files changed

Lines changed: 343 additions & 153 deletions

‎docs/source/runtime-python-api-reference.rst‎

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
Runtime Python API Reference
22
----------------------------------
3-
The Python ``executorch.runtime`` module wraps the C++ ExecuTorch runtime. It can load and execute serialized ``.pte`` program files: see the `Export to ExecuTorch Tutorial <tutorials/export-to-executorch-tutorial.html>`__ for how to convert a PyTorch ``nn.Module`` to an ExecuTorch ``.pte`` program file. Execution accepts and returns ``torch.Tensor`` values, making it a quick way to validate the correctness of the program.
3+
The Python ``executorch.runtime`` module wraps the C++ ExecuTorch runtime. It can load and execute serialized ``.pte`` program files: see the `Export to ExecuTorch Tutorial <tutorials/export-to-executorch-tutorial.html>`__ for how to convert a PyTorch ``nn.Module`` to an ExecuTorch ``.pte`` program file. Execution accepts and returns ``torch.Tensor`` values, making it a quick way to validate the correctness of the program. To run a program without PyTorch installed, see `Without PyTorch`_ below.
44

55
For detailed information on how APIs evolve and the deprecation process, please refer to the `ExecuTorch API Life Cycle and Deprecation Policy <api-life-cycle.html>`__.
66

@@ -21,6 +21,26 @@ For detailed information on how APIs evolve and the deprecation process, please
2121
:members: execute, metadata
2222

2323
.. autoclass:: ExecuTorchResult
24-
:members: shape, strides, dtype, dtype_name, nbytes, to_host, from_host
24+
:members: shape, strides, dtype, device, nbytes, to_host, from_host
2525

26-
Without PyTorch, ``Method.execute`` returns ``ExecuTorchResult`` values: read a CPU result through the buffer protocol, NumPy, or DLPack, read a CUDA result through DLPack or ``to_host()``, and place host data on a CUDA device with ``ExecuTorchResult.from_host()``.
26+
Without PyTorch
27+
~~~~~~~~~~~~~~~
28+
29+
The runtime also works when PyTorch is not installed. Then ``Method.execute`` takes NumPy arrays as inputs and returns ``ExecuTorchResult`` values, which NumPy can read. For a model that adds its two inputs:
30+
31+
.. code-block:: python
32+
33+
import numpy as np
34+
from executorch.runtime import Runtime
35+
36+
method = Runtime.get().load_program("model.pte").load_method("forward")
37+
out = method.execute([np.ones((2, 2), np.float32), np.ones((2, 2), np.float32)])[0]
38+
print(out.dtype) # float32
39+
result = np.asarray(out) # a NumPy view of the output, no copy
40+
print(result.tolist()) # [[2.0, 2.0], [2.0, 2.0]]
41+
42+
Other buffers, such as ``memoryview``, work as inputs too. NumPy has no bfloat16 type, so it reads a bfloat16 result as 16-bit integers that hold the raw bits.
43+
44+
For data on a GPU, use DLPack, the standard way to share tensors between Python libraries. This needs ExecuTorch built with CUDA support. Pass a CuPy array to a method that takes CUDA inputs. Read a CUDA result with ``cupy.from_dlpack(out)``, or copy it to the CPU with ``out.to_host()``. ``ExecuTorchResult.from_host()`` copies CPU data to a CUDA device.
45+
46+
To get ``ExecuTorchResult`` values even when PyTorch is installed, set ``EXECUTORCH_PYBINDINGS_TENSOR_OUTPUT=executorch`` before importing ExecuTorch.

‎extension/pybindings/README.md‎

Lines changed: 24 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,10 @@ separate borrowed view. In torch output mode such a result is a view of memory
6565
torch does not own, so it cannot grow with `resize_()` and cannot be shared
6666
through `torch.multiprocessing`. Call `.clone()` for a normal torch tensor.
6767

68-
`dtype_name` names a result's dtype, including dtypes NumPy lacks, such as
69-
`"bfloat16"`.
68+
An `ExecuTorchResult`'s `dtype` is the dtype's name as torch spells it, such as
69+
`"float32"` or `"bfloat16"`.
70+
`method_meta().output_tensor_meta(i).dtype_name()` gives the same name before
71+
the method runs.
7072

7173
`Method.get_attribute()` returns a view of mutable state. A constant can sit
7274
in read-only memory, so it comes back read-only: NumPy and the buffer protocol
@@ -79,6 +81,16 @@ Calls on a shared method or module are serialized. A sequence of separate
7981
held by the caller around the whole sequence: `set_inputs` hides the outputs
8082
until the next `execute`, so another thread's call can land in between.
8183

84+
A run may reuse the memory of inputs that export planned, or write an input in
85+
place. So each `execute` installs such inputs again from what the caller
86+
passed, or from the copy the bindings made when an input needed converting. A
87+
module does the same for inputs kept from an earlier call. A change to the
88+
storage of a torch tensor or buffer input is then refused; a DLPack producer
89+
must keep its memory in place while the bindings hold it. A result returned
90+
with `clone_outputs=False` lives in the method's output memory, so passing it
91+
back in as an input means each run reads what the run before it wrote there;
92+
pass a copied result instead.
93+
8294
To select `ExecuTorchResult` outputs even when PyTorch is installed, set the
8395
environment variable before importing the bindings. The setting is fixed when
8496
the extension loads, so importing PyTorch later does not change the output type.
@@ -149,11 +161,16 @@ GIL. On an idle device a wait is short:
149161
synchronization has no C++ ATen dependency.
150162

151163
For bfloat16 inputs, raw byte buffers use the shape declared by the exported
152-
method. Bfloat16 results expose flat bytes through the buffer protocol because
153-
NumPy and PEP 3118 do not define that numeric dtype. DLPack consumers receive the
154-
original bfloat16 dtype and tensor shape. A bfloat16 result passed back in is
155-
read through DLPack too, so it keeps its own shape and dim order instead of
156-
taking the method's.
164+
method. NumPy and PEP 3118 do not define bfloat16, so the buffer protocol shows
165+
a bfloat16 result as 16-bit unsigned integers in its real shape and layout.
166+
Other dtypes NumPy lacks, such as float8, are shown the same way, as integers of
167+
the same width. NumPy converts that view as integers, so
168+
`np.asarray(out, dtype=np.float32)` gives the bit patterns as numbers, not the
169+
bfloat16 values. DLPack consumers, such as `torch.from_dlpack(out)`, receive the
170+
original bfloat16 dtype. A bfloat16 result passed back in is read through DLPack
171+
too, so it keeps its dtype, shape, and dim order. A bfloat16 input does not
172+
accept 16-bit integers, so that integer data is never read as bfloat16 by
173+
mistake; pass the result itself or raw bytes.
157174

158175
```python
159176
import cupy as cp

0 commit comments

Comments
 (0)