You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 363be8c
Browse filesBrowse the repository at this point in the historyBrowse files
Support torch.empty_like when lowering to Edge, and speed up GroupBasedPartitioner (#24247)
Fixes found while lowering the SmolVLA robot policy to ExecuTorch.
Commits 5 and 6 change only the partitioner and do not need commits 1 to
4.
**1. Accept symbolic sizes in `_empty_dim_order`.** The op's schema took
`int[] size`. So `torch.empty(x.shape[0], 3)` fixed that dim to its
export size, and the program failed at any other size.
`torch.empty(n.item(), 3)` failed to lower at all. It now takes
`SymInt[] size`.
**2. Give inserted dim order copies their own memory.** A copy that
`LegalizePortableDimOrderPass` inserted after a clone listed in
`reinplace_extra_ops` reused the memory it read from, so values came out
wrong.
**3. Convert the source of a portable copy to the layout of its
destination.** On main, both copies below give wrong values without an
error. The portable `copy.out` kernel ignores the layout of the source,
and treats the destination as contiguous when it broadcasts:
```python
buffer = torch.empty(2, 3, 4, 6, memory_format=torch.channels_last)
buffer.copy_(x) # x is a contiguous (2, 3, 4, 6) tensor
buffer.copy_(y) # y is (3, 4, 6), broadcast to every batch
```
The in-place `copy_` used for buffer write-backs fails at runtime when
the layouts differ. The pass now converts the source to the layout of
the destination, and leaves alone the copies the kernel gets right. Only
contiguous and 4-D channels last destinations are handled, because exir
can convert a tensor only to these two layouts.
**4. Support `torch.empty_like` when lowering to Edge.** Core ATen
decomposition turns it into `aten.empty_permuted`, which is not core
ATen, so `to_edge` fails.
It cannot be decomposed right away, because in Edge its input can have
another layout than in eager: `x.permute(0, 3, 1, 2)` of a contiguous
tensor is channels last in eager, but a contiguous copy in Edge. So a
new `ReplaceEmptyLikePass` lowers it after views have become copies.
Limits:
- Model inputs that are neither contiguous nor 4-D channels last still
fail `to_edge`.
- When `x` is the output of `avg_pool2d` on a channels last tensor,
`to_executorch` later makes `x` contiguous, but `y =
torch.empty_like(x)` stays channels last, so `y + x` fails at runtime.
- If a partitioner preserves `empty_like`, every default-format
`empty_like` is kept, even those its filter rejects, and those fail to
load for lack of a kernel.
**5. Make `GroupBasedPartitioner` cycle checks fast on large graphs.**
It now uses bitmasks instead of per-node sets of all upstream and
downstream nodes. With `XnnpackPartitioner` on SmolVLA, partitioning
went from 123 min to 31 s, and its peak memory from about 30 GB to about
12 GB.
**6. Remove four structures that `GroupBasedPartitioner` fills but never
reads.**
For the same graph, neither partitioner change moves a node to another
partition (checked on thousands of random graphs).
## What was tested
On Linux x86_64 CPU: 12 of the 18 new tests fail on main, and the other
6 check existing behavior. The exir tests, including exir/backend, and
the XNNPACK tests have no new failures.
Co-authored-by: Anthony Shoumikhin <anthony@shoumikh.in>
0 commit comments