Repository navigation
Make nonzero and the Vulkan and WebGPU delegates follow the declared dim order - #24771
shoumikhin wants to merge 11 commits into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24771
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 423 PendingAs of commit e14f332 with merge base 8c3f75d ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@claude /pr-review |
|
Claude finished @shoumikhin's task in 29m 48s —— View job PR Review: #24771Needs discussion: Detailed reviewSummaryThis PR fixes the layout of the
Code Quality
Infrastructure
Testing
API Design
Backward Compatibility
Performance
RecommendationNeeds Discussion The main open question is whether
No code was changed. I did not build or run tests, because running Python here needs approval. Every finding comes from reading the code. All 19 changed files were reviewed. |
abe9f62 to
ad64232
Compare
|
@claude /pr-review |
|
Claude finished @shoumikhin's task in 23m 44s —— View job PR Review: #24771Needs discussion: the Detailed reviewSummaryThis PR makes the portable
The main problem is that the
Backward Compatibility
Code Quality
Testing
RecommendationNeeds Discussion The
I didn't change any code. I didn't build or run tests either, so every finding comes from reading the code. All 27 changed files were reviewed. |
|
@claude /pr-review |
ad64232 to
ee9b608
Compare
|
Claude finished @shoumikhin's task in 33m 34s —— View job PR Review: #24771Needs discussion: the portable Detailed reviewSummaryThe PR makes the portable Code Quality
Testing
Backward Compatibility
Performance
RecommendationNeeds Discussion Before merging, decide which readers that ignore the dim order of the
I didn't change any code. I didn't build or run tests either, so every finding comes from reading the code. The verifier's output-side point is marked low confidence for that reason. All 27 changed files were reviewed. |
ee9b608 to
1fd85e6
Compare
|
Thanks. Changes:
Not changed: a C++ test for the returned external constant (same predicate as the Python test; needs a new generated program), and skipping the copy for 1-D or index-only readers (kernels compare dim orders, not bytes, so |
|
@claude /pr-review |
|
Claude finished @shoumikhin's task in 48m 47s —— View job PR Review: #24771Approve with one follow-up: Vulkan, WebGPU and OpenVINO delegates that take a host Detailed reviewSummaryThe PR makes the portable Backward Compatibility
Performance
Code Quality
RecommendationApprove Nothing blocks. The Vulkan, WebGPU and OpenVINO misread comes from a dim-order gap those runtimes already have, which this PR exposes for Progress
Review only: no code was changed, and nothing was built or run. |
1fd85e6 to
70d8556
Compare
|
Claude finished @shoumikhin's task in 58m 45s —— View job Request Changes: the new strided paths in the portable PR Review: #24771SummaryThe PR copies the result of Code Quality
Testing
Backward Compatibility
RecommendationRequest Changes The crash on tensors without strides blocks approval, because it is a regression in four public kernel paths. Fix the
Nothing was built or run for this review; the findings come from reading the code. |
A delegate whose graph reads and writes every tensor in the default dim order has to copy an input stored in another dim order, such as channels last or the column-major result of `nonzero`, into a buffer in the default order, and copy an output from such a buffer back into its own dim order. The next changes make the Vulkan and WebGPU runtimes do this, so this adds the shared code first. `tensor_data_is_in_default_dim_order()` tells whether the elements of a tensor already sit where the default dim order puts them. Like `output_is_raw_copyable` in the CUDA runtime, it ignores the strides of dims of size 1, so a channels-last tensor with a height and a width of 1 needs no copy. `copy_to_default_dim_order()` and `copy_from_default_dim_order()` copy between a tensor and such a buffer. They visit the elements in the default dim order, with a loop over the last dim and an odometer over the other dims that tracks the offset in the data, so they need no division per element. Elements of 1, 2, 4 or 8 bytes are copied with a fixed-size copy. A tensor whose data is already in the default order is copied with one memcpy. The new tests copy a [2, 3, 4] tensor stored in each of its dim orders to and from a buffer, for elements of 1, 2, 4 and 8 bytes and with the element size known only at run time. They also cover scalars, empty tensors and dim orders that differ only in dims of size 1.
The Vulkan graph reads and writes every tensor as if it were stored in the default dim order, but the runtime copied inputs and outputs as flat bytes. So an input in another dim order, such as a column-major `[N, 2]` index, was read in the wrong places, and an output declared in another dim order, such as channels-last, got its data in the default order, with no error. The runtime now copies such an input into the default order before the upload, and such an output from the default order into its own after the download, with `copy_to_default_dim_order()` and `copy_from_default_dim_order()`. A tensor whose data already sits in the default order, such as any tensor in the default dim order, takes the same path as before, with no extra copy. Before, a channels-last input and output came out right only for elementwise ops on tensors of one shape, because both were copied flat. Now the op does not matter. This also applies to method inputs and outputs that go straight to the delegate. A caller that passes a tensor in the default dim order for such an input has its data treated as already in the method's dim order, as portable kernels always did. Such an output now holds its data in its declared dim order. So a caller that relied on the flat copies must now pass and read the data in the method's dim order. The portable `nonzero` kernel still writes its result row by row, although the program declares dim order `(1, 0)` for it. A Vulkan delegate that reads such a result was right only because neither side followed the dim order, so it reads that result wrong until the kernel writes in its declared order. The new tests pass a column-major index to a delegated `gather`, and scale the channels of a channels-last input. Both fail without the fix.
The WebGPU graph reads and writes every tensor as if it were stored in the default dim order, but the runtime uploaded inputs and read back outputs as flat bytes. So an input in another dim order, such as a column-major `[N, 2]` index, was read in the wrong places, and an output declared in another dim order, such as channels-last, got its data in the default order, with no error. The runtime now copies such an input into the default order before the upload, with `copy_to_default_dim_order()`. It reads such an output back into a buffer in the default order and copies it into its own dim order with `copy_from_default_dim_order()`. A tensor whose data already sits in the default order, such as any tensor in the default dim order, takes the same path as before, with no extra copy. Before, a channels-last input and output came out right only for elementwise ops on tensors of one shape, because both were copied flat. Now the op does not matter. This also applies to method inputs and outputs that go straight to the delegate. A caller that passes a tensor in the default dim order for such an input has its data treated as already in the method's dim order, as portable kernels always did. Such an output now holds its data in its declared dim order. So a caller that relied on the flat copies must now pass and read the data in the method's dim order. The new native tests feed a column-major index to a delegated `gather`, and scale the channels of a channels-last input. Both fail without the fix. A Python test checks that both models are delegated, so the native tests cannot pass on portable kernels instead.
ATen returns the result of `torch.nonzero` column by column: for a result of shape `[N, ndim]`, the strides are `(1, N)`. Exported programs keep that layout, so the output tensor of `nonzero.out` has dim order `(1, 0)`. The portable kernel ignored this and always wrote row by row, so the runtime read the indices back in the wrong places, with no error: ``` x = [[0, 1, 2], [3, 0, 0]] eager: [[0, 1], [0, 2], [1, 0]] ExecuTorch: [[0, 2], [1, 1], [0, 0]] ``` The kernel now writes each index through the strides of the output, so it is right for both layouts. A resized dynamic output gets its strides recomputed from its dim order, so this also holds for the upper-bounded outputs that exported programs use. The kernel also reads its input through its strides now. A `nonzero` that reads another `nonzero` result stays right this way. Before, it read its input row by row, which matched only the old row-by-row writes. This also fixes the `nonzero` of an input whose data really is in another dim order, such as a channels-last mask. The portable `gather` and `scatter` kernels read their tensors as if they were stored row by row, and `scatter` copied `self` into its output byte by byte. With a `nonzero` result as the index, as the `self` of either op, or as the `src` of `scatter`, they were right only because of the old row-by-row writes. They now read the index, `self` and `src` through their strides, as `index` already does for its indices, and `scatter` copies `self` into its output through the strides of both. A contiguous index is still read in order, as before. `unsqueeze_copy` copied its input into its contiguous output byte by byte, so it had the same problem. It now copies an input in another dim order element by element, in logical order, and refuses an output that is not in the default dim order, as `view_copy` already does. `narrow_copy` read its input row by row too. The slice copy behind it now copies element by element, through the strides of both tensors, unless its block copy is right for them: both tensors hold their data in the default dim order, or both have the same dim order and it starts with dims 0 to dim, in order. That holds for any two contiguous tensors, for channels-last tensors whose other dims have size 1, and for two channels-last tensors sliced along dim 0, so those keep the block copy. Two channels-last tensors sliced along another dim used to take the block copy too, so `slice_copy` and `narrow_copy` of a channels-last tensor along C, H or W gave wrong values, and so did a `narrow_copy` into an output in another dim order. They take the strided copy now. The portable and Cadence HiFi `slice_copy` refuse mixed dim orders before they reach that copy, as before. The Cadence Fusion G3 `slice_copy` checks dim orders only when built with OP_ARG_CHECK, so for the dtypes it does not pass to `xa_nn_slice`, such as Long, it now gets the strided copy too, or an error when its output does not have the shape of the slice. So a program where `nonzero` feeds `gather`, `scatter`, `unsqueeze_copy` or `narrow_copy` keeps working, including programs exported before this change and programs exported with `_skip_dim_order=True`. The portable `as_strided_copy` refused an input that is not in the default dim order. Its strides index the storage of its input, whatever the dim order of that input, as in PyTorch, so `as_strided` of a `nonzero` result reads the right elements through them. It now requires only its output to be in the default dim order, since it writes that order. A `nonzero` result now has the `(1, 0)` layout that the program declares. C++, Java or Objective-C code that reads it row by row, and a delegate that takes it as an input, must follow its dim order, as the CUDA, Core ML, Vulkan, WebGPU and OpenVINO backends do. These kernels read any input in its declared dim order now, not only a `nonzero` result. So an input that declares another dim order but holds its data in the default order is read differently than before. A delegate that writes an output in the default order, whatever dim order the output declares, produces such an input. The new kernel tests fail without the fix.
`scatter.src` and `scatter_add` accept a 0-dim index with a 1-dim src, as PyTorch does: ``` torch.scatter(torch.zeros(3), 0, torch.tensor(1), torch.tensor([5., 6., 7.])) # tensor([0., 5., 0.]) ``` The portable kernels find the src element of each index element from the coordinate of that index element. A 0-dim index has no coordinate, so they read the offset in src from uninitialized memory and gave wrong values or crashed, depending on what the stack held. For `scatter.src` this happens with a 0-dim or a 1-dim self, and for `scatter_add` with a 1-dim self. `gather` already handles a 0-dim index on its own, and `scatter.src` and `scatter_add` now do the same: they write, or add, the first element of src at the position the index holds. `scatter.value` does not need this, since it reads no src and its output then has at most one dim. Without this change the new tests read the uninitialized coordinate, so whether they fail depends on the stack; valgrind reports the read.
The Arm partitioner rejects a node with an int64 output unless every user turns the value into another dtype. To check the users, it took the first FakeTensor of each one. A user that reads the size of the tensor, `sym_size`, holds a SymInt instead, so the check raised and the whole lowering failed: ``` TypeError: Expected a FakeTensor in meta["val"] of node sym_size, but got SymInt ``` The next change copies the result of `nonzero` into the default dim order before partitioning, and the size reads of the result then read that int64 copy. Without this change, every model with `nonzero` then fails to lower on every TOSA spec, as `test_torch_functions_tosa_FP` and `test_torch_functions_tosa_INT` with `nonzero` show. The check now skips such users, as the input check of the same class already skips inputs that hold a SymInt. A size read does not read the int64 data.
ATen returns the result of `torch.nonzero` column by column, and exported programs keep that dim order, `(1, 0)`. Most portable kernels read only the default dim order or channels last, so most ops that use the result fail at run time. For example `x[idx[:, 0], idx[:, 1]]` with `idx = torch.nonzero(x)`: ``` Check failed (all_contiguous || all_channels_last): 2 input tensors have different dim orders KernelCall failed at instruction 0:1 in operator aten::select_copy.int_out ``` A copy that keeps the layout of the result, such as `idx.float()`, `idx.clone()` or `idx / 2`, fails earlier, in `to_edge`, because the dim order ops cannot express `(1, 0)`: ``` Failed to map a given dim_order: [1, 0] to a torch.memory_format ``` `to_edge` and `to_edge_transform_and_lower` now copy the result of `nonzero` into the default dim order, before `MemoryFormatOpsPass` and before partitioning. The older `ExirExportedProgram.to_edge`, which `exir.capture` uses, does the same. So every op after it reads the default dim order, every delegate is compiled for the layout it reads at run time, and a caller that reads a returned result row by row reads it right. `as_strided` of the result still reads the result itself, because its strides refer to the layout that ATen gave the result. So a delegate that follows strides, such as OpenVINO, reads it right, and so does the portable `as_strided_copy`, which indexes the storage of its input. `as_strided` of a value computed from the result, such as `idx.float()`, reads that value in the default dim order, where eager keeps the layout of `idx`. The copy moves `N * ndim` integers, which is small next to the `nonzero` scan of the whole input, but it needs its own planned memory, as large as the upper bound of the result, unless a delegate takes it into its partition. A graph without `nonzero` is left as it is, so its export does not pay for retracing the graph. Like `MemoryFormatOpsPass`, this runs only with dim order ops, which are on by default. With `_skip_dim_order=True`, and in programs exported before this change, there is no copy and the result keeps its `(1, 0)` layout. Ops that follow strides, such as `gather`, `scatter`, `unsqueeze_copy`, `narrow_copy` and `index` (as an index), read it right. Most other ops refuse it with an error, as before. The caller must follow its dim order. Exporting again without `_skip_dim_order=True` adds the copy. The new tests fail without this change, except the `as_strided` test, which fails if `as_strided` reads the copy. One runs a program that reads the result through `select_copy` and through `.float()`, and compares it with eager. It also fails without the previous fix to the `nonzero` kernel, since the copy reads the kernel output. It also checks that the copy does not bind the size that depends on the data, as `nonzero` does: a CUDA delegate that takes the copy without `nonzero` fails to lower otherwise. Another checks that a graph without `nonzero` is not retraced. Another exports through `ExirExportedProgram.to_edge` and checks that the returned result is copied and contiguous. Another checks that `as_strided` reads the result itself, runs the program and compares it with eager. A CUDA test runs a program where a CUDA `add` reads the result, another CUDA `add` reads a `mul` of it, and a portable `select` reads it, and compares it with eager. `test_optional_tensor_list` now expects the copy before `index`.
The portable `nonzero` kernel now writes its result in its declared dim order, `(1, 0)`, column by column, and export copies that result into the default dim order. The Qualcomm backend runs `RemoveRedundancy` before partitioning, and it removed every `_clone_dim_order`, so it also removed that copy. Qualcomm has no `nonzero` op, and its partitioner keeps every op that reads a size that depends on the data on the CPU, so `nonzero` and the ops that read its result run on the CPU. Without the copy, such an op, like `idx + 1` or `x[idx[:, 0], idx[:, 1]]`, fails at run time because its inputs have different dim orders, as on main. And a returned result keeps dim order `(1, 0)`, so a caller that reads it row by row, as it could before, reads it wrong. `RemoveRedundancy` now keeps a `_clone_dim_order` that reads a `nonzero` result. It still removes the others, as before. The new test fails without this change.
…rnels `tensor_data_is_in_default_dim_order` read `strides()[dim]` for every dim. A portable tensor can be built without strides, for example `TensorImpl(ScalarType::Float, 2, sizes, data)`, and then `strides()` is empty, so the check read through a null pointer and crashed. The rest of the runtime treats such a tensor as contiguous: `tensor_is_contiguous` and `tensor_is_default_dim_order` both return true for it. The helper now does the same, so `copy_to_default_dim_order` and `copy_from_default_dim_order` take the plain copy for it. The strided paths of the portable `nonzero`, `gather`, `scatter.src` and the slice copy behind `slice_copy` and `narrow_copy` indexed `strides()` or `dim_order()` of such a tensor through a null pointer too, where before they used only its sizes. They now use contiguous strides for a tensor whose data is in the default dim order, which includes a tensor without strides. `scatter` copies `self` with a plain copy when both `self` and `out` hold their data in the default order. A loaded program always sets strides, so only code that calls these kernels directly with such tensors crashed. Also mark the tensor that `copy_from_default_dim_order` writes as `@param[out]`, as the header does for other written parameters. It stays a `const Tensor&`, like the destination of `copy_tensor_data` in the same header, because a const tensor still owns writable data. Drop the `<cstring>` include, which already comes through `dim_order_util.h`. The new tests crash without this change.
Export copies the result of `nonzero` into the default dim order, and `as_strided` of the result itself still reads the result, because its strides refer to the layout that ATen gave it. But `as_strided` of a value computed from the result, like `idx.float()`, read that value in the default dim order, while eager keeps the column-major layout of `idx`. So it read other elements, with no error: ``` x = [[0, 1, 2], [3, 0, 0]] torch.as_strided(torch.nonzero(x).float(), (2, 2), (1, 3)) eager: [[0, 1], [0, 2]] ExecuTorch: [[0, 2], [1, 1]] ``` `NonzeroDimOrderPass` now tracks the values computed from a copy whose layout differs from the one ATen gave them, also through ops with more than one output, like `sort`. It raises a clear error when `as_strided` reads one of them, and asks the caller to call `.contiguous()` on the `nonzero` result or on that value, which gives it the same layout in eager and in the program. Dims of size 1 do not count, since they move no element, so the `(N, 1)` result of a 1-D input still exports. A value whose layout matches eager, like a reduction or a `.contiguous()` copy, still exports too. Values that pass through a `cond` or `map` are not tracked. Views need more care. The edge dialect turns a view, like `idx[1:]` or `idx[0]`, into a new contiguous tensor, while eager reads it through the storage of its input. So a view of a tracked value counts as laid out differently, and so does a value computed from such a view, like `idx[1:] + 0`, unless only one of its dims has more than one element, since the layout that ATen gives it is unknown here. The new tests of refused values fail without this change. Two new tests guard against refusing too much: a reduction with a result size not known at export fails if a single output of `max` counts as relaid, and a sum over a view fails if every value computed from a view counts as relaid.
Export copies the result of `nonzero` into the default dim order before partitioning, so every op and delegate is compiled for the layout it reads. But in the final program the copy needs its own planned buffer, as large as the upper bound of the result: about 12 MiB more for a returned `nonzero` of a 1024 x 1024 input. It also needs the `_clone_dim_order` kernel. And a runtime built before the kernel fix writes the result row by row, so the copy reads it in the wrong places, with no error. The new `FoldNonzeroCopyPass` removes the copy in `to_executorch` when it is the only reader of the result and neither runs in a delegate, which is the usual case. The copy node then runs `nonzero` itself and keeps its own name, which the graph signature can list as an output. So `nonzero` declares the default dim order and its kernel writes that order. Such a program plans one buffer for the result, as before, needs no extra kernel, and gives the right indices on old and new runtimes. The pass runs right after `SpecPropPass`, because every retrace gives `nonzero` the layout of ATen again, and it changes the graph in place. When the copy stays, for example because a delegate takes the copy but not `nonzero`, `nonzero` keeps the `(1, 0)` layout, which its kernel follows. `test_optional_tensor_list` expects the program of main again, without the copy. The new test checks that the program has no copy, that `nonzero` declares the default dim order, that every output in the graph signature names a node of the graph, and that it runs right. It fails without the fold, and also if the fold removes the copy node instead. On a runtime built before the kernel fix, the end-to-end tests of the earlier changes also fail without it.
fcd2767 to
e14f332
Compare
|
@claude review effort=max |
PR Review: #24771
|

What is wrong today
Each tensor declares a dim order (memory layout), but:
nonzerowrites its result row by row, while the program declares it column by column (dim order(1, 0)), as PyTorch returns it. So indices are wrong, and most ops reading them fail.What this change does
nonzero,gather,scatter,unsqueeze_copy,narrow_copyand the slice copy follow strides. This covers the(1, 0)result of older programs, and channels-last slices along C, H or W, also in Cadence HiFi and, for some dtypes, Fusion G3.as_strided_copyaccepts an input in any dim order, since its strides index the storage.scatterandscatter_addwith a 0-dim index no longer read garbage.nonzeroresult into the default dim order before partitioning, so every op and delegate gets the layout it reads. The Qualcomm backend keeps this copy, and the Arm partitioner no longer crashes on it.to_executorchremoves the copy again when it is the only reader of anonzerothat is not delegated.nonzerothen writes the default dim order itself: no extra buffer, and older runtimes return right indices too.as_stridedof a value computed from the result, likeidx.float()oridx[1:], fails at export, since its layout differs from PyTorch. Call.contiguous()on it first.Behavior change
unsqueeze_copyrefuses a non-default output dim order, asview_copydoes (export never makes one).as_stridedalso reads the result, the copy stays, with the_clone_dim_orderkernel and a second buffer.What was tested
On Linux x86_64: C++ kernel and util tests, exir tests, and Arm TOSA lowering of
nonzeromodels on 6 specs. Exported programs ran on the new C++ runtime and an older one. New tests fail without their fix, except the WebGPU Python delegation check. Native WebGPU CI fails on main before its tests, and Cadence CI skips this pull request.Fixes #24763
cc @SS-JIA @manuelcandales @digantdesai @cbilgin @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani