Skip to content

Make nonzero and the Vulkan and WebGPU delegates follow the declared dim order - #24771

Open
shoumikhin wants to merge 11 commits into
pytorch:mainfrom
shoumikhin:nonzero-masked-select-device-io
Open

shoumikhin wants to merge 11 commits into
pytorch:mainfrom
shoumikhin:nonzero-masked-select-device-io

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What is wrong today

Each tensor declares a dim order (memory layout), but:

  • nonzero writes its result row by row, while the program declares it column by column (dim order (1, 0)), as PyTorch returns it. So indices are wrong, and most ops reading them fail.
  • Vulkan and WebGPU copy inputs and outputs as flat bytes, so channels-last or column-major tensors are wrong.

What this change does

  1. Vulkan and WebGPU convert inputs and outputs in another dim order through new runtime helpers.
  2. nonzero, gather, scatter, unsqueeze_copy, narrow_copy and the slice copy follow strides. This covers the (1, 0) result of older programs, and channels-last slices along C, H or W, also in Cadence HiFi and, for some dtypes, Fusion G3. as_strided_copy accepts an input in any dim order, since its strides index the storage. scatter and scatter_add with a 0-dim index no longer read garbage.
  3. Export copies the nonzero result into the default dim order before partitioning, so every op and delegate gets the layout it reads. The Qualcomm backend keeps this copy, and the Arm partitioner no longer crashes on it.
  4. to_executorch removes the copy again when it is the only reader of a nonzero that is not delegated. nonzero then writes the default dim order itself: no extra buffer, and older runtimes return right indices too.
  5. as_strided of a value computed from the result, like idx.float() or idx[1:], fails at export, since its layout differs from PyTorch. Call .contiguous() on it first.

Behavior change

  • These delegates and kernels now follow the declared dim order. Callers that pass or read NCHW bytes for a tensor declared in another order must change, as must delegates that write default-order data into such outputs.
  • unsqueeze_copy refuses a non-default output dim order, as view_copy does (export never makes one).
  • When the copy runs inside a delegate, or as_strided also reads the result, the copy stays, with the _clone_dim_order kernel and a second buffer.

What was tested

On Linux x86_64: C++ kernel and util tests, exir tests, and Arm TOSA lowering of nonzero models on 6 specs. Exported programs ran on the new C++ runtime and an older one. New tests fail without their fix, except the WebGPU Python delegation check. Native WebGPU CI fails on main before its tests, and Cadence CI skips this pull request.

Fixes #24763

cc @SS-JIA @manuelcandales @digantdesai @cbilgin @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani

@shoumikhin shoumikhin added the release notes: exir Changes to any dialects and passes on these dialects, such as memory planning label Nov 7, 2026
Copilot AI balanced review requested due to automatic review settings November 7, 2026 05:38
@pytorch-bot

pytorch-bot Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24771

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 423 Pending

As of commit e14f332 with merge base 8c3f75d (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 7, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@shoumikhin

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Claude finished @shoumikhin's task in 29m 48s —— View job


PR Review: #24771

Needs discussion: MemoryPlanningPass now raises for device inputs that it could leave unplanned, the nonzero kernel fix silently changes what existing .pte files return to callers that ignore dim order, and several behaviors that the PR promises have no test.

Detailed review

Summary

This PR fixes the layout of the nonzero output, lets masked_select through the edge verifier, refuses planned device inputs at export, serializes GPU constants from a host copy, and treats external constants that a method returns as memory planned. It has three main problems:

  • The new device-input check raises where planning could just skip those inputs.
  • The nonzero kernel fix changes what existing .pte files return to callers that ignore dim order.
  • Several claimed behaviors have no test.

Code Quality

  • exir/emit/_emitter.py:473-475: a constant with no host data now fails with PyTorch's error instead of ExecuTorch's. The removed guard raised an ExecuTorch message that named the device and the fix. Now a meta constant fails inside storage.cpu() with PyTorch's own copy error, wrapped in InternalError. test_emit_meta_constant_raises (exir/emit/test/test_emit.py:1862) matches PyTorch's wording ("meta tensor"), so it breaks if PyTorch rewords that error. Keep an explicit check for storage.device.type == "meta" that raises an ExecuTorch message, and assert on that message. Fix this →

Infrastructure

  • exir/passes/BUCK:240-254: the memory_planning_pass target is missing //executorch/exir:schema. memory_planning_pass.py:32 now imports DeviceType from executorch.exir.schema, which resolves only through //executorch/exir:tensor. Commit 2 (Copy the nonzero result into the default dim order for portable kernels) adds a direct dep for its own new import, so add one here too. Fix this →

Testing

  • Commit 6 (Stop overwriting external constants that a method returns): the new InvalidState return has no test. The commit message lists two changes that callers can see:

    • set_output_data_ptr now returns InvalidState for an output that is an external constant, through the existing check at runtime/executor/method.cpp:1626-1633.
    • is_memory_planned() is now true for such an output.

    The tests cover these only in part:

    • ExternalConstantAttributesAreMemoryPlanned (runtime/executor/test/method_meta_test.cpp:207-220) only checks attribute_tensor_meta, and its ModuleAddMul program returns no constant. No C++ test checks output_tensor_meta(i).is_memory_planned() or the InvalidState return.
    • test_program_data_separation_returns_constant (extension/pybindings/test/test_pybindings.py:3075-3091) checks the output change only through the bindings. It runs once and returns only the constant. So it misses the rest of the bug the commit message describes: "every read of the constant sees zeros, including operators that use it to compute other outputs". A model that returns self.c, x + self.c and runs twice would cover it. Fix this →
  • The two masked_select limitations in the description have no test. The description says: "Inputs with 2 elements or fewer still fail in to_executorch(), and an input that broadcasts to a bigger shape can fail at run time, both with an error." test_masked_select (exir/program/test/test_program.py:439-458) only uses a 2×3 input with a mask of the same shape, so neither error is checked. When shapes are static, the export could also refuse the broadcast case outright. Fix this →

  • The description's claim about GPU models with no backend has no test. It says that such a model "now exports, with the same program as a CPU export". The only new test (backends/cuda/tests/test_cuda_export.py:1091-1118) lowers through the CUDA partitioner. To cover the claim, add a CUDA-only test that exports the GPU and CPU versions of one model, with no partitioner, and compares their to_executorch().buffer. Fix this →

API Design

  • exir/passes/memory_planning_pass.py:143-167 (called at 338-339): the pass raises for device inputs that it could leave unplanned. The pass knows which method inputs stay on a device, and the runtime refuses to fill a planned input in device memory. Even so, it raises and asks for MemoryPlanningPass(alloc_graph_input=False), which applies to the whole method:

    • CPU inputs stop being planned too, and the method then uses the caller's memory for them. test_skip_h2d_partial_with_intermediate_input (exir/tests/test_propagate_device_pass.py:1815) had to drop its check that the CPU input a is planned in CPU memory, because that setup can no longer export.
    • Users still have to pair the two settings by hand. This PR had to change alloc_graph_input in ten test configs in that file.

    Memory planning already special-cases device-resident constants (exir/memory_planning.py:442 and :553). When non-CPU planning is on, collect_specs_from_nodes could likewise skip the specs of method inputs whose device != DeviceType.CPU. The default MemoryPlanningPass() would then work with skip_h2d_for_method_inputs=True. Verifier.verify_graph_input_output (exir/memory_planning.py:300-315) compares input allocation with alloc_graph_input, so it would need the same exception. Fix this →

Backward Compatibility

  • Commit 1 (Write nonzero indices in the dim order of the output tensor) changes what existing programs return to callers that ignore dim order. Programs without the new copy record dim order (1, 0) for the nonzero output. The kernel now writes that output column by column (kernels/portable/cpu/op_nonzero.cpp:69-79). For inputs with 2 or more dims, this layout differs from the row-by-row data the kernel wrote before. Two groups of programs have no copy:

    • every .pte exported before this PR
    • new exports with _skip_dim_order=True, where LegalizePortableDimOrderPass does not run (exir/program/_program.py:807-808)

    Consumers that read through the strides get the fix. index.Tensor is one of them. A C++ caller that reads the output's const_data_ptr<int64_t>() row by row without checking dim_order() is in the opposite position. It used to get the right indices by accident, and it now gets transposed ones with no error. Call this out in the release notes.

Performance

  • exir/passes/legalize_portable_dim_order_pass.py:143-155: the copy is added even when the consumer reads through strides. In test_optional_tensor_list (exir/emit/test/test_emit.py:667-668), the program now copies between nonzero and index.Tensor. But index.Tensor reads its indices through their strides (kernels/portable/cpu/util/advanced_index_util.cpp:139-140). Each copy adds a kernel call and a second buffer the size of nonzero's upper bound (numel × ndim × 8 bytes), and both buffers are live at the same time. Commit 1 already makes the kernel follow the output strides, so giving the nonzero output itself the default dim order might remove the copy. Low confidence: a later retrace may bring back ATen's strides, which may be why you chose the copy.

Recommendation

Needs Discussion

The main open question is whether MemoryPlanningPass should leave device inputs unplanned instead of raising (see API Design). That choice sets the user-facing contract for skip_h2d_for_method_inputs, so it is best settled before this lands. The nonzero kernel change for existing .pte files needs at least a release note. The other findings are minor and could go in a follow-up.


  • Gather context (diff, commits, CLAUDE.md, checklist)
  • Run three independent reviewers
  • Consolidate findings
  • Fact-check findings
  • Post review

No code was changed. I did not build or run tests, because running Python here needs approval. Every finding comes from reading the code. All 19 changed files were reviewed.

@shoumikhin
shoumikhin force-pushed the nonzero-masked-select-device-io branch from abe9f62 to ad64232 Compare November 7, 2026 18:04
@shoumikhin shoumikhin changed the title Fix nonzero, masked_select, GPU constants and returned external constants, and refuse device inputs that cannot run Fix nonzero, masked_select, GPU constants and returned external constants, and leave device inputs unplanned Nov 7, 2026
Copilot AI balanced review requested due to automatic review settings November 7, 2026 18:04
@shoumikhin

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@claude

claude Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Claude finished @shoumikhin's task in 23m 44s —— View job


PR Review: #24771

Needs discussion: the nonzero kernel fix silently changes the layout that C++ callers read from any program exported with _skip_dim_order=True, a setting many in-tree export paths hard-code; the other five findings are minor.

Detailed review

Summary

This PR makes the portable nonzero kernel write through its output strides and copies the result into the default dim order. It also:

  • accepts masked_select in the edge verifier
  • leaves method inputs that stay on a device unplanned
  • serializes GPU constants from a host copy
  • counts external constants as memory planned

The main problem is that the nonzero layout change affects every program exported with _skip_dim_order=True, not only programs exported before this PR. The other findings are minor:

  • an error message that no longer matches its cause
  • a weaker memory planning check
  • two small issues in the new code
  • masked_select limits with no test

Backward Compatibility

  • kernels/portable/cpu/op_nonzero.cpp:69-83: the new layout reaches every program exported with _skip_dim_order=True, and many in-tree export paths hard-code that setting.
    • LegalizePortableDimOrderPass, which adds the new copy, runs only when _skip_dim_order is false (exir/program/_program.py:807-808). Without the copy, the kernel now writes the (1, 0) output column by column.
    • A C++ caller that reads const_data_ptr<int64_t>() row by row got the right indices before this PR. Now it reads them in the wrong order, with no error. A delegate that ignores dim order has the same problem.
    • The description tells these users to re-export without _skip_dim_order. Many in-tree flows can't do that without a code change, because they hard-code the setting:
      • extension/export_util/utils.py:20-23: the default config of export_to_edge and export_to_exec_prog, which examples/portable/scripts/export.py uses
      • the backend configs in backends/cadence/aot/compiler.py:299, backends/samsung/utils/export_utils.py:27, backends/native/__init__.py:23 and backends/apple/coreai/__init__.py:59
      • examples/xnnpack/aot_compiler.py:114, examples/cuda/scripts/export.py:30, examples/apple/coreml/scripts/export.py:40, and many model export scripts under examples/models/
    • Either add the nonzero copy in this mode too, or announce the change as BC-breaking in the release notes. The copy would have to be _to_dim_order_copy, because the portable _to_copy kernel refuses to change dim order (kernels/portable/cpu/op_to_copy.cpp:50-52).

Code Quality

  • exir/emit/_emitter.py:489-499: the error no longer states why it fires.

    • The message still says that "constants serialized into the program must be in CPU memory". That is no longer the rule, because the emitter now serializes other GPU constants from a host copy.
    • The error now has two causes: an operator outside any delegate reads the constant, or the storage is meta and has no data. The message names neither.
    • It also does not name the operator that reads the constant, so the user can't tell which op to delegate. _read_on_host could return the first such user for the message to name, as PropagateDevicePass does with first_reader.name (exir/passes/propagate_device_pass.py:588-591).

    Fix this →

  • exir/emit/_emitter.py:223: copy ops are matched by namespace through getattr.

    • getattr(user.target, "namespace", None) != "et_copy" is the dynamic getattr pattern that CLAUDE.md asks to avoid.
    • It would also exempt any op added to et_copy later.
    • Compare against torch.ops.et_copy._h2d_copy.out and _d2h_copy.out explicitly, as exir/tests/test_propagate_device_pass.py:95-97 does.

    Fix this →

  • exir/memory_planning.py:286-292: the verifier stops checking host inputs when a device input comes last.

    • In verify_graph_input_output, the last placeholder decides graph_input_allocated.
    • A device input now sets that flag to self.alloc_graph_input, so the final assert compares that value with itself.
    • Take a method shaped like forward(a_host, b_device), the shape test_skip_h2d_partial_with_intermediate_input (exir/tests/test_propagate_device_pass.py:1809) exports. If a is planned wrongly, the verifier no longer catches it.
    • Set the flag from a skipped placeholder only when it is still None.

    Fix this →

  • exir/passes/legalize_portable_dim_order_pass.py:101-104: _nonzero_results is never cleared.

    • The dict holds every copy and its result, with their FX nodes and fake tensors, for as long as the pass instance lives.
    • to_executorch creates a new instance for each method. But edge.transform([LegalizePortableDimOrderPass()]) reuses one instance across all methods of a program.
    • Clear the dict at the start of call().

    Fix this →

Testing

  • exir/program/test/test_program.py:439-458: the two masked_select limits in the description have no test.

    • The description says that two cases fail with an error, never with wrong values:
      • an input that broadcasts to a bigger mask
      • an input of 2 elements or fewer
    • test_masked_select only uses a 2×3 input with a mask of the same shape.
    • A test that asserts each error would catch a PyTorch change to the fake masked_select bound that turns either case into wrong output.

    Fix this →

Recommendation

Needs Discussion

The nonzero layout change should either reach _skip_dim_order=True programs through the copy, or be announced as BC-breaking, before this lands. The other findings are minor and can be fixed here or in a follow-up.


  • Gather context (diff, commits, CLAUDE.md, checklist)
  • Run three independent reviewers
  • Consolidate findings
  • Fact-check findings
  • Post review

I didn't change any code. I didn't build or run tests either, so every finding comes from reading the code. All 27 changed files were reviewed.

@shoumikhin

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@shoumikhin
shoumikhin force-pushed the nonzero-masked-select-device-io branch from ad64232 to ee9b608 Compare November 7, 2026 19:16
Copilot AI balanced review requested due to automatic review settings November 7, 2026 19:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@claude

claude Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Claude finished @shoumikhin's task in 33m 34s —— View job


PR Review: #24771

Needs discussion: the portable nonzero kernel now writes its result column by column, as its dim order says, so any reader that ignores the dim order gets transposed indices with no error — a delegate in a new export, or gather's self and scatter's src in an older or _skip_dim_order=True program; the other findings are minor.

Detailed review

Summary

The PR makes the portable nonzero kernel write through the output strides and copies its result into the default dim order for portable kernels, lets masked_select through the edge dialect check, serializes a GPU constant from a host copy when no host operator reads it, keeps external constants that a method returns from being overwritten, and leaves method inputs that stay on a device unplanned. The main problem is that readers that ignore the dim order of the nonzero result now read transposed indices with no error. The rest are test gaps, a copy that is often unneeded, and small emitter and verifier problems.

Code Quality

  • exir/emit/_emitter.py:223: _read_on_host finds the copy ops with getattr(user.target, "namespace", None) != "et_copy". CLAUDE.md asks for no dynamic getattr, and this matches any present or future et_copy op. exir/passes/propagate_device_pass.py:251 and :382 compare the et_copy targets explicitly. The emitter sees the out variants, so compare with those, for example torch.ops.et_copy._h2d_copy.out. Fix this →
  • exir/emit/_emitter.py:489-499: the error no longer matches the rule it enforces. It still says constants "must be in CPU memory", but device constants are now accepted when no host operator reads them. For meta storage, "Move them to CPU" is the wrong advice, because there is no data to move. The message also doesn't name the host operator that reads the constant, although _read_on_host finds that operator and then returns only a bool. Fix this →
  • exir/memory_planning.py:288-291: in verify_graph_input_output, a placeholder with no specs left after filtering now sets graph_input_allocated = self.alloc_graph_input. The last placeholder decides the final value. So for forward(a_host, b_device), the check at lines 318-320 compares alloc_graph_input with itself, and a wrong allocation of a_host passes. Set the flag only while it is still None. Low confidence, not run: the old branch also set graph_output_allocated for parameter placeholders. A method with parameters whose output has no tensor specs (for example, one that returns only a SymInt) therefore passed the assert at line 322 by accident, and now fails it with "graph_output_allocated not set". If such methods are supported, this is also a BC break. Fix this →

Testing

  • backends/cuda/tests/test_cuda_export.py:1118-1120: test_buffer_returned_outside_the_delegate returns early in fbcode before edge.to_executorch() (line 1126). So in fbcode it never runs the emitter path this PR changes, although only the runtime is missing there. Call to_executorch() before the guard. Fix this →
  • runtime/executor/test/method_meta_test.cpp:207-220: ExternalConstantAttributesAreMemoryPlanned checks only the attributes of add_mul_program. Nothing tests the case the commit fixes: a returned external constant should report is_memory_planned() in its output meta, and set_output_data_ptr should reject it with InvalidState (runtime/executor/method.cpp:1626-1633). Fix this →
  • exir/program/test/test_program.py:439-458: test_masked_select only uses a 2×3 input with a mask of the same shape. Commit a45ddcd documents two limits, and neither is tested: an input with 2 elements or fewer, which to_executorch() rejects, and an input that broadcasts to a bigger mask and selects more elements than it has, which fails at run time. The commit promises these stop "with an error, never with wrong values", so a test should pin that. Fix this →
  • backends/cuda/tests/test_device_placement.py:125-143: test_gpu_constant_read_by_a_delegate exports through load(), which uses the default device config, so the constant reaches the delegate through _h2d_copy. No test covers the executorch_call_delegate exemption in _read_on_host (exir/emit/_emitter.py:222), where a delegate reads a GPU constant directly. That needs a module still on the GPU, exported with PropagateDeviceConfig(skip_h2d_for_constants=True). Fix this →

Backward Compatibility

  • kernels/portable/cpu/op_nonzero.cpp:69-80: the kernel now writes through the output strides. Export gives nonzero the (1, 0) dim order, so the result is now written column by column, where it used to be row by row. Any reader that ignores the dim order now reads transposed indices, with no error:
    • Delegates in new exports. call_delegate (exir/passes/legalize_portable_dim_order_pass.py:148-153) gives a delegate the original (1, 0) result instead of the copy. It assumes the delegate honours the dim order it was compiled for (docstring, lines 86-89). A delegate that treats its inputs as contiguous was correct before this PR and is silently wrong now. The only delegate test (exir/tests/test_legalize_portable_dim_order_pass.py, with CpuOnlyPartitioner) checks the graph wiring and runs nothing. Consider inserting the copy before partitioning instead, in _generate_edge_program next to convert_constant_dim_order_pass (exir/program/_program.py:831-838), whose comment states the same goal: "so a delegate sees the layout the program stores". Delegates would then compile for the default dim order. That also removes the call_delegate override and the _nonzero_results dict (lines 101-104), which is never cleared. The cost is that partitioners see the copy. Fix this →
    • Programs exported before this PR, and _skip_dim_order=True programs. These get no copy. gather reads self (kernels/portable/cpu/op_gather.cpp:57) and scatter reads src (kernels/portable/cpu/op_scatter.cpp:51) with coordinateToIndex, which is row-major, and neither kernel checks the dim order. So torch.gather(torch.nonzero(x), 1, idx) returned the right values before this PR and returns wrong ones now. Commit 9a878c2 says that in these programs "gather, scatter and index read it right, the other ops refuse it with an error", but that holds only where the result is the index. Read self and src through their strides with linearize_access_indexes, as the kernels now read the index. A tensor_is_default_dim_order check would also reject [N, 1] results, whose bytes are the same in both layouts. Fix this →

Performance

  • exir/passes/legalize_portable_dim_order_pass.py:155-164: the copy follows every nonzero, including when no reader needs it. A reader that uses the result only as an index already follows the strides: index.Tensor, and now the index of gather and scatter. exir/emit/test/test_emit.py:667-668 now expects a copy before index.Tensor. The copy also runs for a 1-D input, whose [N, 1] result has the same bytes in both layouts. Each copy costs an extra kernel call and a second upper-bound buffer of numel × ndim × 8 bytes, live at the same time as the result. That is about 25 MB for a [1, 3, 512, 512] input. Skip the copy in those cases. Fix this →

Recommendation

Needs Discussion

Before merging, decide which readers that ignore the dim order of the nonzero result must keep working. gather's self and scatter's src in programs without the copy have a small, concrete fix that should land in this PR. For delegates that treat their inputs as contiguous, either move the copy before partitioning or state the contract that delegates must honour the dim order. The other findings are minor.

  • Read the diff and the PR context
  • Run three independent reviews
  • Merge duplicate findings
  • Fact-check findings
  • Post the review

I didn't change any code. I didn't build or run tests either, so every finding comes from reading the code. The verifier's output-side point is marked low confidence for that reason. All 27 changed files were reviewed.

@shoumikhin
shoumikhin force-pushed the nonzero-masked-select-device-io branch from ee9b608 to 1fd85e6 Compare November 8, 2026 00:39
Copilot AI balanced review requested due to automatic review settings November 8, 2026 00:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@shoumikhin

Copy link
Copy Markdown
Contributor Author

Thanks. Changes:

  • Kernels: gather and scatter read self and src through strides, and scatter copies self in its output's dim order. unsqueeze_copy had the same bug and nonzero misread its own input, so both follow strides. Old and _skip_dim_order=True programs where nonzero feeds them work; narrow_copy refuses such input.
  • Delegates: the copy stays in to_executorch. Moving it does not help runtimes that ignore input dim order (Vulkan claims and drops it), and Qualcomm's to_edge transform would delete it before partitioning. The description states the contract; a CUDA test runs a delegate reading the result. The dict is per pass instance, one per method.
  • Verifier: the input flag is set only while unset; a method returning only .item() now passes.
  • Emitter: explicit et_copy targets; separate messages for meta storage and a named host op.
  • Tests: masked_select limits, the delegate case with skip_h2d_for_constants, to_executorch() before the is_fbcode() return.

Not changed: a C++ test for the returned external constant (same predicate as the Python test; needs a new generated program), and skipping the copy for 1-D or index-only readers (kernels compare dim orders, not bytes, so [N, 1] results need it; index-only use is rare).

@shoumikhin

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Claude finished @shoumikhin's task in 48m 47s —— View job


PR Review: #24771

Approve with one follow-up: Vulkan, WebGPU and OpenVINO delegates that take a host nonzero result as input now return wrong results with no error, and the other three findings are minor.

Detailed review

Summary

The PR makes the portable nonzero kernel write the (1, 0) dim order the program declares, and copies the result to the default dim order for portable kernels. It also lets masked_select pass the edge dialect check, serializes GPU constants from a host copy, leaves device method inputs unplanned, and stops overwriting external constants that a method returns. There is one backward-compatibility regression in in-tree delegates and three minor issues. All 37 changed files were reviewed.

Backward Compatibility

  • Delegates that ignore dim order now silently misread a host nonzero result.
    • op_nonzero.cpp:70-84 now writes column by column, as the (1, 0) dim order says, and LegalizePortableDimOrderPass.call_delegate (legalize_portable_dim_order_pass.py:148) gives a delegate the raw result, not the copy.
    • The d6facc1 commit message says the Vulkan, WebGPU and OpenVINO runtimes read every input in the default dim order, so they "now misread such an input". For example, VulkanBackend.cpp:855-859 copies numel() elements from const_data_ptr() without checking the dim order.
    • Before this PR the kernel wrote row by row, so those delegates got the right answer by accident. Now they get wrong indices and report no error.
    • The same silent change hits C++ callers that read a returned nonzero result row by row, in programs exported earlier or with _skip_dim_order=True.
    • This is separate from where the copy lives, which the thread settled. The ask is to make the misread fail loudly. Those runtimes could return an error when tensor_is_default_dim_order (runtime/core/exec_aten/util/tensor_util.h:1283) is false for an input, or their partitioners could decline such inputs. At minimum, link tracking issues for the three backends from the PR. Fix this →

Performance

  • nonzero now does ndim multiply-adds per input element, even when the input is contiguous. op_nonzero.cpp:77-81 rebuilds in_index from every stride on each iteration, where the old loop read in_data[i]. For the usual contiguous mask in_index == i, so the extra work is pure overhead on a scan of the whole input. Read in_data[i] when tensor_is_default_dim_order(input) holds, and keep the strided read for other dim orders. Fix this →

Code Quality

  • In the recommended config, a method that returns its device input turns off the memory planning check.
    • The PropagateDeviceConfig docstring now recommends MemoryPlanningPass(alloc_graph_output=False) with the default alloc_graph_input=True (propagate_device_config.py:41).
    • With skip_d2h_for_method_outputs, the output reads a device input placeholder directly (propagate_device_pass.py:583). Its spec is then in get_graph_output_tensors, and also in get_graph_input_tensors, which skips device-resident constants (memory_planning.py:457) but not device inputs.
    • verify_graph_input_output then returns early at memory_planning.py:249-253 and checks none of the method's inputs or outputs.
    • Skip meta["device_input"] placeholders in get_graph_input_tensors too. collect_specs_from_nodes already skips their specs at line 561, so planning doesn't change. Fix this →
  • The _d2h_copy.out exemption in _host_reader (_emitter.py:230) can never match. PropagateDevicePass creates a _d2h_copy only for non-constant method inputs (propagate_device_pass.py:425-434, 598-601) and for delegate outputs (_insert_d2h_for_getitem, 641-642). No _d2h_copy ever reads a constant, so the entry describes a reader that doesn't exist. Drop it. Fix this →

Recommendation

Approve

Nothing blocks. The Vulkan, WebGPU and OpenVINO misread comes from a dim-order gap those runtimes already have, which this PR exposes for nonzero results. It shouldn't stay silent, so add a guard or tracking issues. The other findings are minor.

Progress

  • Read the diff and the PR context
  • Run three independent reviews
  • Merge duplicate findings
  • Fact-check findings
  • Post the review

Review only: no code was changed, and nothing was built or run.

@shoumikhin
shoumikhin force-pushed the nonzero-masked-select-device-io branch from 1fd85e6 to 70d8556 Compare November 8, 2026 11:17
@shoumikhin
shoumikhin requested a review from SS-JIA as a code owner November 8, 2026 11:17
Copilot AI balanced review requested due to automatic review settings November 8, 2026 11:17
@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @shoumikhin's task in 58m 45s —— View job


Request Changes: the new strided paths in the portable nonzero, gather, scatter and compute_slice code dereference a null pointer for tensors built without strides, which worked before this PR, and as_strided of a nonzero result still exports into a program that the portable runtime refuses.

PR Review: #24771

Summary

The PR copies the result of nonzero into the default dim order before partitioning, removes that copy again where nonzero can write the default order itself, and makes several portable kernels and the Vulkan and WebGPU backends follow the dim order of their tensors. Two issues need a fix: a crash on tensors without strides, and an as_strided path that exports but can't run on the portable kernels. The other two findings are minor.

Code Quality

  • as_strided of the nonzero result exports, then fails at run time.
    • exir/passes/memory_format_ops_pass.py:112-116 points as_strided_copy at the raw nonzero result. Because of that, FoldNonzeroCopyPass doesn't fold it, and nonzero keeps ATen's dim order (1, 0).
    • kernels/portable/cpu/op_as_strided_copy.cpp:40-43 returns InvalidArgument for any input that isn't in the default dim order.
    • Result: torch.as_strided(torch.nonzero(x), (2, 2), (1, 3)) fails when the program runs, as it does on main.
    • test_as_strided_reads_nonzero_result_itself (exir/tests/test_memory_format_ops_pass.py:562-576) checks only the Edge graph. Every exir test that runs a program folds the copy, so no test runs this path.
    • Fix: as_strided reads the storage of its input, so the kernel only needs out in the default dim order. Relax those two checks to that, or refuse this case at export like the other as_strided cases. Then run the program in that test.
    • I found this by reading the code; I didn't run it.
    • Fix this →
  • _is_view_copy looks up the view op by name.
    • exir/passes/memory_format_ops_pass.py:240-252 builds the view op's name from the copy's name and looks it up with getattr on torch.ops.aten and on the packet. CLAUDE.md rules out dynamic getattr patterns.
    • The view_copy tag answers the same question. torchgen asserts it on the _copy variant of each view op, and exir/verification/verifier.py:180 already uses it.
    • Fix: line 130 becomes view = isinstance(op, EdgeOpOverload) and torch.Tag.view_copy in op._op.tags and self._is_relaid(args[0]). That removes the helper and the get_view_copy_of_view_op import.
    • Fix this →

Testing

  • The new WebGPU tests Index.ColumnMajorGatherIndex and DimOrder.ChannelsLastInputAndOutput (in backends/webgpu/test/native/test_index.cpp) run only in test-webgpu-native.yml. That workflow runs nightly, on push to main, and on ciflow/nightly/* tags, so no pull request job checks the reorder added to WebGPUBackend.cpp against them. Adding the ciflow/nightly label would run them before merge.

Backward Compatibility

  • The new strided paths crash on tensors without strides.
    • These lines index strides() or dim_order() without checking that they exist:
      • kernels/portable/cpu/op_nonzero.cpp:70-71 (out)
      • kernels/portable/cpu/op_gather.cpp:62 (in)
      • kernels/portable/cpu/op_scatter.cpp:82 (src)
      • kernels/portable/cpu/util/slice_util.cpp:195-198 (in and out), plus lines 223 and 228 when only one of the two has strides
    • Trigger: a lean-mode TensorImpl built without them, such as TensorImpl(ScalarType::Float, 2, sizes, data), returns empty ArrayRefs. ArrayRef::operator[] doesn't check bounds, so passing such a tensor to nonzero_out, gather_out, scatter_src_out or slice_copy_Tensor_out reads through a null pointer.
    • Before this PR, these kernels used only sizes (coordinateToIndex, contiguous writes, block copies), so those calls worked.
    • The runtime treats such tensors as valid:
      • the TensorImpl constructor defaults both pointers to null;
      • internal_resize_contiguous skips missing strides (runtime/core/portable_type/tensor_impl.cpp:163);
      • this PR's tensor_data_is_in_default_dim_order returns true for them, and CopyWithDefaultDimOrderOfTensorWithoutStrides tests exactly this tensor.
    • A loaded Method can't hit this, because the tensor parser always sets both fields (runtime/executor/tensor_parser_portable.cpp:74-76,134-148). Callers that invoke these kernels directly can.
    • Fix: take the contiguous path whenever tensor_data_is_in_default_dim_order(t) is true. In compute_slice, check that before the dim order prefix loop, and use contiguous strides in the element loop for a tensor without strides. Add lean-mode kernel tests with such tensors.
    • Fix this →

Recommendation

Request Changes

The crash on tensors without strides blocks approval, because it is a regression in four public kernel paths. Fix the as_strided path in the same round: the redirect added for it doesn't work on the portable runtime. Both fixes are small.


  • Read the diff
  • Run three independent reviews
  • Merge duplicate findings
  • Fact-check findings
  • Post the review

Nothing was built or run for this review; the findings come from reading the code.

A delegate whose graph reads and writes every tensor in the default dim
order has to copy an input stored in another dim order, such as channels
last or the column-major result of `nonzero`, into a buffer in the
default order, and copy an output from such a buffer back into its own
dim order. The next changes make the Vulkan and WebGPU
runtimes do this, so this adds the shared code first.

`tensor_data_is_in_default_dim_order()` tells whether the elements of a
tensor already sit where the default dim order puts them. Like
`output_is_raw_copyable` in the CUDA runtime, it ignores the strides of
dims of size 1, so a channels-last tensor with a height and a width of 1
needs no copy.

`copy_to_default_dim_order()` and `copy_from_default_dim_order()` copy
between a tensor and such a buffer. They visit the elements in the
default dim order, with a loop over the last dim and an odometer over
the other dims that tracks the offset in the data, so they need no
division per element. Elements of 1, 2, 4 or 8 bytes are copied with a
fixed-size copy. A tensor whose data is already in the default order is
copied with one memcpy.

The new tests copy a [2, 3, 4] tensor stored in each of its dim orders
to and from a buffer, for elements of 1, 2, 4 and 8 bytes and with the
element size known only at run time. They also cover scalars, empty
tensors and dim orders that differ only in dims of size 1.
The Vulkan graph reads and writes every tensor as if it were stored in
the default dim order, but the runtime copied inputs and outputs as flat
bytes. So an input in another dim order, such as a column-major `[N, 2]`
index, was read in the wrong places, and an output declared in another
dim order, such as channels-last, got its data in the default order,
with no error.

The runtime now copies such an input into the default order before the
upload, and such an output from the default order into its own after the
download, with `copy_to_default_dim_order()` and
`copy_from_default_dim_order()`. A tensor whose data already sits in the
default order, such as any tensor in the default dim order, takes the
same path as before, with no extra copy. Before, a channels-last input
and output came out right only for elementwise ops on tensors of one
shape, because both were copied flat. Now the op does not matter.

This also applies to method inputs and outputs that go straight to the
delegate. A caller that passes a tensor in the default dim order for
such an input has its data treated as already in the method's dim order,
as portable kernels always did. Such an output now holds its data in its
declared dim order. So a caller that relied on the flat copies must now
pass and read the data in the method's dim order.

The portable `nonzero` kernel still writes its result row by row,
although the program declares dim order `(1, 0)` for it. A Vulkan
delegate that reads such a result was right only because neither side
followed the dim order, so it reads that result wrong until the kernel
writes in its declared order.

The new tests pass a column-major index to a delegated `gather`, and
scale the channels of a channels-last input. Both fail without the fix.
The WebGPU graph reads and writes every tensor as if it were stored in
the default dim order, but the runtime uploaded inputs and read back
outputs as flat bytes. So an input in another dim order, such as a
column-major `[N, 2]` index, was read in the wrong places, and an output
declared in another dim order, such as channels-last, got its data in
the default order, with no error.

The runtime now copies such an input into the default order before the
upload, with `copy_to_default_dim_order()`. It reads such an output back
into a buffer in the default order and copies it into its own dim order
with `copy_from_default_dim_order()`. A tensor whose data already sits
in the default order, such as any tensor in the default dim order, takes
the same path as before, with no extra copy. Before, a channels-last
input and output came out right only for elementwise ops on tensors of
one shape, because both were copied flat. Now the op does not matter.

This also applies to method inputs and outputs that go straight to the
delegate. A caller that passes a tensor in the default dim order for
such an input has its data treated as already in the method's dim order,
as portable kernels always did. Such an output now holds its data in its
declared dim order. So a caller that relied on the flat copies must now
pass and read the data in the method's dim order.

The new native tests feed a column-major index to a delegated `gather`,
and scale the channels of a channels-last input. Both fail without the
fix. A Python test checks that both models are delegated, so the native
tests cannot pass on portable kernels instead.
ATen returns the result of `torch.nonzero` column by column: for a
result of shape `[N, ndim]`, the strides are `(1, N)`. Exported programs
keep that layout, so the output tensor of `nonzero.out` has dim order
`(1, 0)`. The portable kernel ignored this and always wrote row by row,
so the runtime read the indices back in the wrong places, with no error:

```
x = [[0, 1, 2], [3, 0, 0]]
eager:      [[0, 1], [0, 2], [1, 0]]
ExecuTorch: [[0, 2], [1, 1], [0, 0]]
```

The kernel now writes each index through the strides of the output, so
it is right for both layouts. A resized dynamic output gets its strides
recomputed from its dim order, so this also holds for the upper-bounded
outputs that exported programs use. The kernel also reads its input
through its strides now. A `nonzero` that reads another `nonzero` result
stays right this way. Before, it read its input row by row, which
matched only the old row-by-row writes. This also fixes the `nonzero` of
an input whose data really is in another dim order, such as a
channels-last mask.

The portable `gather` and `scatter` kernels read their tensors as if
they were stored row by row, and `scatter` copied `self` into its output
byte by byte. With a `nonzero` result as the index, as the `self` of
either op, or as the `src` of `scatter`, they were right only because of
the old row-by-row writes. They now read the index, `self` and `src`
through their strides, as `index` already does for its indices, and
`scatter` copies `self` into its output through the strides of both. A
contiguous index is still read in order, as before. `unsqueeze_copy`
copied its input into its contiguous output byte by byte, so it had the
same problem. It now copies an input in another dim order element by
element, in logical order, and refuses an output that is not in the
default dim order, as `view_copy` already does. `narrow_copy` read its
input row by row too. The slice copy behind it now copies element by
element, through the strides of both tensors, unless its block copy is
right for them: both tensors hold their data in the default dim order,
or both have the same dim order and it starts with dims 0 to dim, in
order. That holds for any two contiguous tensors, for channels-last
tensors whose other dims have size 1, and for two channels-last tensors
sliced along dim 0, so those keep the block copy. Two channels-last
tensors sliced along another dim used to take the block copy too, so `slice_copy` and `narrow_copy` of a
channels-last tensor along C, H or W gave wrong values, and so did a
`narrow_copy` into an output in another dim order. They take the strided
copy now. The portable and Cadence HiFi `slice_copy` refuse mixed dim
orders before they reach that copy, as before. The Cadence Fusion G3
`slice_copy` checks dim orders only when built with OP_ARG_CHECK, so for
the dtypes it does not pass to `xa_nn_slice`, such as Long, it now gets
the strided copy too, or an error when its output does not have the
shape of the slice. So a program where `nonzero` feeds `gather`,
`scatter`, `unsqueeze_copy` or `narrow_copy` keeps working, including
programs exported before this change and programs exported with
`_skip_dim_order=True`.

The portable `as_strided_copy` refused an input that is not in the
default dim order. Its strides index the storage of its input, whatever
the dim order of that input, as in PyTorch, so `as_strided` of a
`nonzero` result reads the right elements through them. It now requires
only its output to be in the default dim order, since it writes that
order.

A `nonzero` result now has the `(1, 0)` layout that the program
declares. C++, Java or Objective-C code that reads it row by row, and a
delegate that takes it as an input, must follow its dim order, as the
CUDA, Core ML, Vulkan, WebGPU and OpenVINO backends do.

These kernels read any input in its declared dim order now, not only a
`nonzero` result. So an input that declares another dim order but holds
its data in the default order is read differently than before. A
delegate that writes an output in the default order, whatever dim order
the output declares, produces such an input.

The new kernel tests fail without the fix.
`scatter.src` and `scatter_add` accept a 0-dim index with a 1-dim src,
as PyTorch does:

```
torch.scatter(torch.zeros(3), 0, torch.tensor(1), torch.tensor([5., 6., 7.]))
# tensor([0., 5., 0.])
```

The portable kernels find the src element of each index element from the
coordinate of that index element. A 0-dim index has no coordinate, so
they read the offset in src from uninitialized memory and gave wrong
values or crashed, depending on what the stack held. For `scatter.src`
this happens with a 0-dim or a 1-dim self, and for `scatter_add` with a
1-dim self. `gather` already handles a 0-dim index on its own, and
`scatter.src` and `scatter_add` now do the same: they write, or add, the
first element of src at the position the index holds. `scatter.value`
does not need this, since it reads no src and its output then has at
most one dim.

Without this change the new tests read the uninitialized coordinate, so
whether they fail depends on the stack; valgrind reports the read.
The Arm partitioner rejects a node with an int64 output unless every
user turns the value into another dtype. To check the users, it took the
first FakeTensor of each one. A user that reads the size of the tensor,
`sym_size`, holds a SymInt instead, so the check raised and the whole
lowering failed:

```
TypeError: Expected a FakeTensor in meta["val"] of node sym_size, but got SymInt
```

The next change copies the result of `nonzero` into the default dim
order before partitioning, and the size reads of the result then read
that int64 copy. Without this change, every model with `nonzero` then
fails to lower on every TOSA spec, as `test_torch_functions_tosa_FP`
and `test_torch_functions_tosa_INT` with `nonzero` show.

The check now skips such users, as the input check of the same class
already skips inputs that hold a SymInt. A size read does not read the
int64 data.
ATen returns the result of `torch.nonzero` column by column, and
exported programs keep that dim order, `(1, 0)`. Most portable kernels
read only the default dim order or channels last, so most ops that use
the result fail at run time. For example `x[idx[:, 0], idx[:, 1]]` with
`idx = torch.nonzero(x)`:

```
Check failed (all_contiguous || all_channels_last): 2 input tensors have
different dim orders
KernelCall failed at instruction 0:1 in operator aten::select_copy.int_out
```

A copy that keeps the layout of the result, such as `idx.float()`,
`idx.clone()` or `idx / 2`, fails earlier, in `to_edge`, because the dim
order ops cannot express `(1, 0)`:

```
Failed to map a given dim_order: [1, 0] to a torch.memory_format
```

`to_edge` and `to_edge_transform_and_lower` now copy the result of
`nonzero` into the default dim order, before `MemoryFormatOpsPass` and
before partitioning. The older `ExirExportedProgram.to_edge`, which
`exir.capture` uses, does the same. So every op after it reads the
default dim order, every delegate is compiled for the layout it reads at
run time, and a caller that reads a returned result row by row reads it
right. `as_strided` of the result still reads the result itself, because
its strides refer to the layout that ATen gave the result. So a delegate
that follows strides, such as OpenVINO, reads it right, and so does the
portable `as_strided_copy`, which indexes the storage of its input. `as_strided` of a value computed from the
result, such as `idx.float()`, reads that value in the default dim
order, where eager keeps the layout of `idx`.

The copy moves `N * ndim` integers, which is small next to the `nonzero`
scan of the whole input, but it needs its own planned memory, as large
as the upper bound of the result, unless a delegate takes it into its
partition. A graph without `nonzero` is left as it is, so its export
does not pay for retracing the graph.

Like `MemoryFormatOpsPass`, this runs only with dim order ops, which are
on by default. With `_skip_dim_order=True`, and in programs exported
before this change, there is no copy and the result keeps its `(1, 0)`
layout. Ops that follow strides, such as `gather`, `scatter`,
`unsqueeze_copy`, `narrow_copy` and `index` (as an index), read it
right. Most other ops refuse it with an error, as before. The caller
must follow its dim order. Exporting again without
`_skip_dim_order=True` adds the copy.

The new tests fail without this change, except the `as_strided` test,
which fails if `as_strided` reads the copy. One runs a program that
reads the result through `select_copy` and through `.float()`, and
compares it with eager. It also fails without the previous fix to the
`nonzero` kernel, since the copy reads the kernel output. It also checks
that the copy does not bind the size that depends on the data, as
`nonzero` does: a CUDA delegate that takes the copy without `nonzero`
fails to lower otherwise. Another checks that a graph without `nonzero`
is not retraced. Another exports through `ExirExportedProgram.to_edge`
and checks that the returned result is copied and contiguous. Another
checks that `as_strided` reads the result itself, runs the program and
compares it with eager. A CUDA test runs a
program where a CUDA `add` reads the result, another CUDA `add` reads a
`mul` of it, and a portable `select` reads it, and compares it with
eager. `test_optional_tensor_list` now expects the copy before `index`.
The portable `nonzero` kernel now writes its result in its declared dim
order, `(1, 0)`, column by column, and export copies that result into
the default dim order. The Qualcomm backend runs `RemoveRedundancy`
before partitioning, and it removed every `_clone_dim_order`, so it also
removed that copy. Qualcomm has no `nonzero` op, and its partitioner
keeps every op that reads a size that depends on the data on the CPU, so
`nonzero` and the ops that read its result run on the CPU. Without the
copy, such an op, like `idx + 1` or `x[idx[:, 0], idx[:, 1]]`, fails at
run time because its inputs have different dim orders, as on main. And a
returned result keeps dim order `(1, 0)`, so a caller that reads it row
by row, as it could before, reads it wrong.

`RemoveRedundancy` now keeps a `_clone_dim_order` that reads a `nonzero`
result. It still removes the others, as before.

The new test fails without this change.
…rnels

`tensor_data_is_in_default_dim_order` read `strides()[dim]` for every
dim. A portable tensor can be built without strides, for example
`TensorImpl(ScalarType::Float, 2, sizes, data)`, and then `strides()` is
empty, so the check read through a null pointer and crashed. The rest of
the runtime treats such a tensor as contiguous: `tensor_is_contiguous`
and `tensor_is_default_dim_order` both return true for it. The helper now
does the same, so `copy_to_default_dim_order` and
`copy_from_default_dim_order` take the plain copy for it.

The strided paths of the portable `nonzero`, `gather`, `scatter.src`
and the slice copy behind `slice_copy` and `narrow_copy` indexed
`strides()` or `dim_order()` of such a tensor through a null pointer
too, where before they used only its sizes. They now use contiguous
strides for a tensor whose data is in the default dim order, which
includes a tensor without strides. `scatter` copies `self` with a plain
copy when both `self` and `out` hold their data in the default order. A
loaded program always sets strides, so only code that calls these
kernels directly with such tensors crashed.

Also mark the tensor that `copy_from_default_dim_order` writes as
`@param[out]`, as the header does for other written parameters. It stays
a `const Tensor&`, like the destination of `copy_tensor_data` in the
same header, because a const tensor still owns writable data. Drop the
`<cstring>` include, which already comes through `dim_order_util.h`.

The new tests crash without this change.
Export copies the result of `nonzero` into the default dim order, and
`as_strided` of the result itself still reads the result, because its
strides refer to the layout that ATen gave it. But `as_strided` of a
value computed from the result, like `idx.float()`, read that value in
the default dim order, while eager keeps the column-major layout of
`idx`. So it read other elements, with no error:

```
x = [[0, 1, 2], [3, 0, 0]]
torch.as_strided(torch.nonzero(x).float(), (2, 2), (1, 3))
eager:      [[0, 1], [0, 2]]
ExecuTorch: [[0, 2], [1, 1]]
```

`NonzeroDimOrderPass` now tracks the values computed from a copy whose
layout differs from the one ATen gave them, also through ops with more
than one output, like `sort`. It raises a clear error when `as_strided`
reads one of them, and asks the caller to call `.contiguous()` on the
`nonzero` result or on that value, which gives it the same layout in
eager and in the program. Dims of size 1 do not count, since they move
no element, so the `(N, 1)` result of a 1-D input still exports. A
value whose layout matches eager, like a reduction or a `.contiguous()`
copy, still exports too. Values that pass through a `cond` or `map` are
not tracked.

Views need more care. The edge dialect turns a view, like `idx[1:]` or
`idx[0]`, into a new contiguous tensor, while eager reads it through
the storage of its input. So a view of a tracked value counts as laid
out differently, and so does a value computed from such a view, like
`idx[1:] + 0`, unless only one of its dims has more than one element,
since the layout that ATen gives it is unknown here.

The new tests of refused values fail without this change. Two new
tests guard against refusing too much: a reduction with a result size
not known at export fails if a single output of `max` counts as relaid,
and a sum over a view fails if every value computed from a view counts
as relaid.
Export copies the result of `nonzero` into the default dim order before
partitioning, so every op and delegate is compiled for the layout it
reads. But in the final program the copy needs its own planned buffer,
as large as the upper bound of the result: about 12 MiB more for a
returned `nonzero` of a 1024 x 1024 input. It also needs the
`_clone_dim_order` kernel. And a runtime built before the kernel fix
writes the result row by row, so the copy reads it in the wrong places,
with no error.

The new `FoldNonzeroCopyPass` removes the copy in `to_executorch` when
it is the only reader of the result and neither runs in a delegate,
which is the usual case. The copy node then runs `nonzero` itself and
keeps its own name, which the graph signature can list as an output.
So `nonzero` declares the default dim order and its kernel writes that
order. Such a program plans one buffer for the result, as before, needs
no extra kernel, and gives the right indices on old and new runtimes.
The pass runs right after `SpecPropPass`, because every retrace gives
`nonzero` the layout of ATen again, and it changes the graph in place.
When the copy stays, for example because a delegate takes the copy but
not `nonzero`, `nonzero` keeps the `(1, 0)` layout, which its kernel
follows.

`test_optional_tensor_list` expects the program of main again, without
the copy.

The new test checks that the program has no copy, that `nonzero`
declares the default dim order, that every output in the graph
signature names a node of the graph, and that it runs right. It fails
without the fold, and also if the fold removes the copy node instead. On a runtime built before the kernel fix, the
end-to-end tests of the earlier changes also fail without it.
@shoumikhin
shoumikhin force-pushed the nonzero-masked-select-device-io branch from fcd2767 to e14f332 Compare November 21, 2026 17:36
Copilot AI balanced review requested due to automatic review settings November 21, 2026 17:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@shoumikhin

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

PR Review: #24771

  • Read the diff and the PR context (11 commits, 35 files)
  • Run three independent reviews
  • Merge duplicate findings
  • Fact-check findings
  • Post the review

View job run

This branch is being deployed

1 in progress deployment
cadence — e14f3321 Deployed Nov 21, 2026 by pytorch-bot[bot] via hifi5s-build / hifi5s #39973
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/nightly ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: arm Issues related to arm backend module: vulkan Issues related to the Vulkan delegate and code under backends/vulkan/ release notes: backends

Projects

None yet

Development

Successfully merging this pull request may close these issues.

torch.nonzero returns wrong indices, and indexing with its result fails at run time

3 participants