Add FuseConcatPass to eliminate redundant concat ops (#18827) by ryan-monroe · Pull Request #18827 · pytorch/executorch

ryan-monroe · 2026-04-11T00:06:22Z

Summary:

Adds an FX-level pass that eliminates concat ops which can be proven structurally redundant before the TOSA backend / Vela compiler ever sees them. In the Gen2 Executorch ARM / Ethos-U stack, torch.cat lowers to TOSA CONCAT, which Vela converts to N MemoryCopy ops — real DMA on the NPU. Catching the obvious cases up front keeps the TOSA flatbuffer fed to Vela smaller, keeps debug graphs honest, and provides defensive coverage on TOSA targets where Vela's own scheduler doesn't run (e.g., the VGF backend).

Five rewrite patterns are handled (inspired by Espresso's bolt/nn/espresso/transforms/remove_nops.py):

Single-input concat: cat([x], dim) ≡ x — replace cat with x.
Concat-then-slice (exact): cat([a, b, ...], dim) feeding a slice_copy that extracts exactly one original input — replace the slice with the corresponding cat input directly.
Slice-then-concat (full): cat([slice(x, d, s0, e0), slice(x, d, s1, e1), ...], dim) reconstructing x exactly (contiguous slices covering the full source dimension) — replace cat with x.
Concat-then-sub-slice: a slice_copy whose range falls entirely within one cat input — replace with an adjusted slice on that input directly.
Slice-then-concat (partial): contiguous slices of the same tensor concatenated back but covering only a sub-range of the source — replace with a single slice on the source.

Empirical impact across the production EMG model fleet

Measured by running every frl/ctrl/torchstream/torchstream/pt2/tests/test_emg_lowering_* quantize+lower test with FuseConcatPass instrumented to log per-call counters, then comparing against the same target with the pass commented out in arm_pass_manager.py. All 8 model targets pass under both configurations.

Model	Cats scanned	Eliminated	Pattern fired
cascade_classifier	5	3 (60%)	single-input
mux_fusion	8	3 (38%)	single-input
combined_control	11	3 (27%)	single-input
cascade_detector	14	0	—
cascade_hw_classifier	12	0	—
handwriting	106	0	—
wake	11	0	—
auth	6	0	—
Total	173	9	all single-input

Two findings worth highlighting

Patterns 2–5 (the slice-related rewrites) never matched on any production EMG model. PyTorch's Aten lowering on this fleet doesn't produce the cat↔slice algebra these patterns target. They remain useful for non-EMG TOSA workloads — and for the VGF backend where Vela's own optimizer doesn't run — but on the current EMG production set they are unexercised.

Vela already folds single-input cats during compilation. A before/after measurement on cascade_classifier (the model with the highest hit rate, 3/5 cats eliminated) shows Vela emits the same 9 MemoryCopy ops and consumes the same 481,339 NPU cycles either way. The eliminated cats reappear in the Vela operator table as Reshape → MemoryCopy instead of Concat → MemoryCopy. Total NPU runtime is unchanged. Pre-Vela artifacts do shrink (TOSA flatbuffer −16 KB / −0.68%, peak staging −1.5 KB / −0.45%), but post-Vela on-device performance is identical.

Net effect

This pass is value-additive even where it doesn't move NPU cycles:

Cleaner TOSA fed into Vela (~16 KB smaller per cascade_classifier instance).
Slightly tighter peak staging during Vela scheduling (~1.5 KB).
Defensive coverage for TOSA-only targets without a Vela-grade scheduler (notably the VGF / Vulkan path).
More truthful FX / EXIR debug graphs — concats that were genuinely no-ops no longer show up in model-explorer, delegation_metadata.json, or the lowered graph dumps.

It does not produce measurable NPU cycle savings on the current EMG production fleet. The patterns that would have produced real Vela savings (cat↔slice algebra) don't appear in these models.

Authored with Claude.

Differential Revision: D97667069

pytorch-bot · 2026-04-11T00:06:27Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18827

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures

As of commit 5cc8286 with merge base ada8e35 ():

NEW FAILURES - The following jobs have failed:

pull / unittest-editable / macos / macos-job (gh)
extension/llm/tokenizers/test/test_hf_tokenizer.py::TestHfTokenizer::test_llama3_2_1b_special_toks
trunk / test-huggingface-transformers-xnnpack (smollm3-3b|xnnpack|--quantize) / linux-job (gh)
RuntimeError: Command docker exec -t a8ae1e747b279ef792307adee2c9302a9f71193ee00202410b5e465fd861a5e2 /exec failed with exit code 1
trunk / test-torchao-huggingface-checkpoints (qwen3_4b, linux.2xlarge, executorch-ubuntu-22.04-clang12, x... / linux-job (gh)
RuntimeError: Command docker exec -t e678bd277cc9a39d6e6f625e45be17a834ba763f584c127b0db00ce39021e208 /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

meta-codesync · 2026-04-11T00:06:32Z

@ryan-monroe has exported this pull request. If you are a Meta employee, you can view the originating Diff in D97667069.

github-actions · 2026-04-11T00:07:13Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Summary: Concat (torch.cat) in the Gen2 Executorch ARM/Ethos-U stack is lowered to TOSA CONCAT, which Vela then converts to N x MemoryCopy operations — real DMA data movement on the NPU. This pass eliminates concat operations that can be proven unnecessary at the FX graph level, preventing Vela from generating MemoryCopy ops entirely. Inspired by Espresso's concat elimination techniques (bolt/nn/espresso/transforms/remove_nops.py), three patterns are handled: 1. Single-input concat: cat([x]) is a no-op, replaced with x. 2. Concat-then-slice: if every consumer of cat([a, b, ...]) is a slice_copy that extracts exactly one original input, bypass both. 3. Slice-then-concat: if contiguous slices of the same tensor are concatenated back, the result is the original tensor. Differential Revision: D97667069

ryan-monroe · 2026-04-14T20:17:56Z

@pytorchbot rerun -f

pytorch-bot · 2026-04-14T20:17:59Z

❌ 🤖 pytorchbot command failed:

@pytorchbot: error: argument command: invalid choice: 'rerun' (choose from 'merge', 'revert', 'rebase', 'label', 'drci', 'lint', 'fix-lint', 'apply-lint', 'cherry-pick')

usage: @pytorchbot [-h]
                   
                   {merge,revert,rebase,label,drci,lint,fix-lint,apply-lint,cherry-pick}
                   ...

Try @pytorchbot --help for more info.

Summary: Concat (torch.cat) in the Gen2 Executorch ARM/Ethos-U stack is lowered to TOSA CONCAT, which Vela then converts to N x MemoryCopy operations — real DMA data movement on the NPU. This pass eliminates concat operations that can be proven unnecessary at the FX graph level, preventing Vela from generating MemoryCopy ops entirely. Inspired by Espresso's concat elimination techniques (bolt/nn/espresso/transforms/remove_nops.py), three patterns are handled: 1. Single-input concat: cat([x]) is a no-op, replaced with x. 2. Concat-then-slice: if every consumer of cat([a, b, ...]) is a slice_copy that extracts exactly one original input, bypass both. 3. Slice-then-concat: if contiguous slices of the same tensor are concatenated back, the result is the original tensor. Differential Revision: D97667069

pytorch-bot · 2026-04-17T21:12:40Z

~~Workflows were awaiting approval.~~ CI has now been triggered for the ciflow labels on this PR.

Summary: Pull Request resolved: pytorch#18827 Concat (torch.cat) in the Gen2 Executorch ARM/Ethos-U stack is lowered to TOSA CONCAT, which Vela then converts to N x MemoryCopy operations — real DMA data movement on the NPU. This pass eliminates concat operations that can be proven unnecessary at the FX graph level, preventing Vela from generating MemoryCopy ops entirely. Inspired by Espresso's concat elimination techniques (bolt/nn/espresso/transforms/remove_nops.py), three patterns are handled: 1. Single-input concat: cat([x]) is a no-op, replaced with x. 2. Concat-then-slice: if every consumer of cat([a, b, ...]) is a slice_copy that extracts exactly one original input, bypass both. 3. Slice-then-concat: if contiguous slices of the same tensor are concatenated back, the result is the original tensor. Differential Revision: D97667069

Ninja91 · 2026-04-21T17:29:46Z

+    return first_start, expected_start
+
+
+class FuseConcatPass(ArmPass):


@gggekov Here's another optimization pass we plan to add. Please comment if this will impact downstream passes/regor transforms

I recommend testing the passes using EthosU85PipelineINT or EthosU55PipelineINT in order to know if the pass works with the current version of Vela.

Overall, for me it makes sense for the optimization of some CONCATs to be done in ExecuTorch rather than Vela.

Summary: Concat (torch.cat) in the Gen2 Executorch ARM/Ethos-U stack is lowered to TOSA CONCAT, which Vela then converts to N x MemoryCopy operations — real DMA data movement on the NPU. This pass eliminates concat operations that can be proven unnecessary at the FX graph level, preventing Vela from generating MemoryCopy ops entirely. Inspired by Espresso's concat elimination techniques (bolt/nn/espresso/transforms/remove_nops.py), three patterns are handled: 1. Single-input concat: cat([x]) is a no-op, replaced with x. 2. Concat-then-slice: if every consumer of cat([a, b, ...]) is a slice_copy that extracts exactly one original input, bypass both. 3. Slice-then-concat: if contiguous slices of the same tensor are concatenated back, the result is the original tensor. Differential Revision: D97667069

gggekov · 2026-04-22T17:46:20Z

Helo @ryan-monroe ,
Thanks for the patch, great initiative to reduce the number of memcpy from CONCATs!

I see in the unit tests, you only test with the executorch.backends.arm.test.tester.test_pipeline.PassPipeline. Given that the end metric you care for is number of CONCATs in the TOSA flatbuffer, wouldn't it make sense to test instead either with the EthosU85PipelineINT / EthosU55PipelineINT or TosaPipelineINT as these pipelines will generate a TOSA fb that is run on device? For counting the number of CONCAT ops in the TOSA fb, you can use the count_tosa_ops in backends/arm/test/tester/test_pipeline.py. See example how to do that here - https://github.com/pytorch/executorch/blob/main/backends/arm/test/misc/test_transpose_counts.py#L537

oscarandersson8218 · 2026-04-23T06:21:38Z

+    rank = len(get_first_fake_tensor(node).shape)
+    dim = _int_arg(node, 1, 0)
+    dim = (dim + rank) % rank
+    start = _int_arg(node, 2, 0)


Don't you need to normalize both start and end as these can be negative?

Would this be negative even when we export with static shapes? @oscarandersson8218

Yes, I would think so as you can do x[-3:3].

Summary: Pull Request resolved: pytorch#18827 Concat (torch.cat) in the Gen2 Executorch ARM/Ethos-U stack is lowered to TOSA CONCAT, which Vela then converts to N x MemoryCopy operations — real DMA data movement on the NPU. This pass eliminates concat operations that can be proven unnecessary at the FX graph level, preventing Vela from generating MemoryCopy ops entirely. Inspired by Espresso's concat elimination techniques (bolt/nn/espresso/transforms/remove_nops.py), three patterns are handled: 1. Single-input concat: cat([x]) is a no-op, replaced with x. 2. Concat-then-slice: if every consumer of cat([a, b, ...]) is a slice_copy that extracts exactly one original input, bypass both. 3. Slice-then-concat: if contiguous slices of the same tensor are concatenated back, the result is the original tensor. Differential Revision: D97667069

Summary: Concat (torch.cat) in the Gen2 Executorch ARM/Ethos-U stack is lowered to TOSA CONCAT, which Vela then converts to N x MemoryCopy operations — real DMA data movement on the NPU. This pass eliminates concat operations that can be proven unnecessary at the FX graph level, preventing Vela from generating MemoryCopy ops entirely. Inspired by Espresso's concat elimination techniques (bolt/nn/espresso/transforms/remove_nops.py), three patterns are handled: 1. Single-input concat: cat([x]) is a no-op, replaced with x. 2. Concat-then-slice: if every consumer of cat([a, b, ...]) is a slice_copy that extracts exactly one original input, bypass both. 3. Slice-then-concat: if contiguous slices of the same tensor are concatenated back, the result is the original tensor. Differential Revision: D97667069

Summary: Pull Request resolved: pytorch#18827 Concat (torch.cat) in the Gen2 Executorch ARM/Ethos-U stack is lowered to TOSA CONCAT, which Vela then converts to N x MemoryCopy operations — real DMA data movement on the NPU. This pass eliminates concat operations that can be proven unnecessary at the FX graph level, preventing Vela from generating MemoryCopy ops entirely. Inspired by Espresso's concat elimination techniques (bolt/nn/espresso/transforms/remove_nops.py), three patterns are handled: 1. Single-input concat: cat([x]) is a no-op, replaced with x. 2. Concat-then-slice: if every consumer of cat([a, b, ...]) is a slice_copy that extracts exactly one original input, bypass both. 3. Slice-then-concat: if contiguous slices of the same tensor are concatenated back, the result is the original tensor. Differential Revision: D97667069

Summary: Concat (torch.cat) in the Gen2 Executorch ARM/Ethos-U stack is lowered to TOSA CONCAT, which Vela then converts to N x MemoryCopy operations — real DMA data movement on the NPU. This pass eliminates concat operations that can be proven unnecessary at the FX graph level, preventing Vela from generating MemoryCopy ops entirely. Inspired by Espresso's concat elimination techniques (bolt/nn/espresso/transforms/remove_nops.py), three patterns are handled: 1. Single-input concat: cat([x]) is a no-op, replaced with x. 2. Concat-then-slice: if every consumer of cat([a, b, ...]) is a slice_copy that extracts exactly one original input, bypass both. 3. Slice-then-concat: if contiguous slices of the same tensor are concatenated back, the result is the original tensor. Differential Revision: D97667069

Summary: Adds an FX-level pass that eliminates concat ops which can be proven structurally redundant before the TOSA backend / Vela compiler ever sees them. In the Gen2 Executorch ARM / Ethos-U stack, `torch.cat` lowers to TOSA `CONCAT`, which Vela converts to N `MemoryCopy` ops — real DMA on the NPU. Catching the obvious cases up front keeps the TOSA flatbuffer fed to Vela smaller, keeps debug graphs honest, and provides defensive coverage on TOSA targets where Vela's own scheduler doesn't run (e.g., the VGF backend). Five rewrite patterns are handled (inspired by Espresso's `bolt/nn/espresso/transforms/remove_nops.py`): 1. **Single-input concat**: `cat([x], dim) ≡ x` — replace cat with x. 2. **Concat-then-slice (exact)**: `cat([a, b, ...], dim)` feeding a `slice_copy` that extracts exactly one original input — replace the slice with the corresponding cat input directly. 3. **Slice-then-concat (full)**: `cat([slice(x, d, s0, e0), slice(x, d, s1, e1), ...], dim)` reconstructing x exactly (contiguous slices covering the full source dimension) — replace cat with x. 4. **Concat-then-sub-slice**: a `slice_copy` whose range falls entirely within one cat input — replace with an adjusted slice on that input directly. 5. **Slice-then-concat (partial)**: contiguous slices of the same tensor concatenated back but covering only a sub-range of the source — replace with a single slice on the source. ## Empirical impact across the production EMG model fleet Measured by running every `frl/ctrl/torchstream/torchstream/pt2/tests/test_emg_lowering_*` quantize+lower test with FuseConcatPass instrumented to log per-call counters, then comparing against the same target with the pass commented out in `arm_pass_manager.py`. All 8 model targets pass under both configurations. | Model | Cats scanned | Eliminated | Pattern fired | | --- | --- | --- | --- | | cascade_classifier | 5 | **3 (60%)** | single-input | | mux_fusion | 8 | **3 (38%)** | single-input | | combined_control | 11 | **3 (27%)** | single-input | | cascade_detector | 14 | 0 | — | | cascade_hw_classifier | 12 | 0 | — | | handwriting | 106 | 0 | — | | wake | 11 | 0 | — | | auth | 6 | 0 | — | | **Total** | **173** | **9** | all single-input | ## Two findings worth highlighting **Patterns 2–5 (the slice-related rewrites) never matched on any production EMG model.** PyTorch's Aten lowering on this fleet doesn't produce the cat↔slice algebra these patterns target. They remain useful for non-EMG TOSA workloads — and for the VGF backend where Vela's own optimizer doesn't run — but on the current EMG production set they are unexercised. **Vela already folds single-input cats during compilation.** A before/after measurement on cascade_classifier (the model with the highest hit rate, 3/5 cats eliminated) shows Vela emits the same 9 `MemoryCopy` ops and consumes the same 481,339 NPU cycles either way. The eliminated cats reappear in the Vela operator table as `Reshape → MemoryCopy` instead of `Concat → MemoryCopy`. Total NPU runtime is unchanged. Pre-Vela artifacts do shrink (TOSA flatbuffer −16 KB / −0.68%, peak staging −1.5 KB / −0.45%), but post-Vela on-device performance is identical. ## Net effect This pass is value-additive even where it doesn't move NPU cycles: - Cleaner TOSA fed into Vela (~16 KB smaller per cascade_classifier instance). - Slightly tighter peak staging during Vela scheduling (~1.5 KB). - Defensive coverage for TOSA-only targets without a Vela-grade scheduler (notably the VGF / Vulkan path). - More truthful FX / EXIR debug graphs — concats that were genuinely no-ops no longer show up in `model-explorer`, `delegation_metadata.json`, or the lowered graph dumps. It does **not** produce measurable NPU cycle savings on the current EMG production fleet. The patterns that would have produced real Vela savings (cat↔slice algebra) don't appear in these models. Authored with Claude. Differential Revision: D97667069

ryan-monroe requested a review from digantdesai as a code owner April 11, 2026 00:06

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 11, 2026

meta-codesync Bot added fb-exported meta-exported labels Apr 11, 2026

meta-codesync Bot changed the title ~~Add FuseConcatPass to eliminate redundant concat ops~~ Add FuseConcatPass to eliminate redundant concat ops (#18827) Apr 13, 2026

ryan-monroe force-pushed the export-D97667069 branch from f91806e to 44c7409 Compare April 13, 2026 19:11

ryan-monroe force-pushed the export-D97667069 branch from 44c7409 to ca6843d Compare April 14, 2026 17:23

ryan-monroe force-pushed the export-D97667069 branch from ca6843d to e14d429 Compare April 14, 2026 18:19

ryan-monroe force-pushed the export-D97667069 branch from e14d429 to 4056395 Compare April 14, 2026 18:25

ryan-monroe force-pushed the export-D97667069 branch from 4056395 to 78f3c89 Compare April 16, 2026 20:15

Ninja91 added partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm ciflow/trunk labels Apr 17, 2026

Ninja91 requested review from Ninja91 and gggekov April 17, 2026 21:13

zingo changed the title ~~Add FuseConcatPass to eliminate redundant concat ops (#18827)~~ Arm backend: Add FuseConcatPass to eliminate redundant concat ops (#18827) Apr 19, 2026

ryan-monroe force-pushed the export-D97667069 branch from 78f3c89 to 55eddbf Compare April 20, 2026 21:59

meta-codesync Bot changed the title ~~Arm backend: Add FuseConcatPass to eliminate redundant concat ops (#18827)~~ Add FuseConcatPass to eliminate redundant concat ops (#18827) Apr 20, 2026

ryan-monroe force-pushed the export-D97667069 branch from 55eddbf to eb266d8 Compare April 21, 2026 03:45

Ninja91 reviewed Apr 21, 2026

View reviewed changes

ryan-monroe force-pushed the export-D97667069 branch from eb266d8 to db25b64 Compare April 22, 2026 01:13

github-actions Bot added the module: arm Issues related to arm backend label Apr 22, 2026

oscarandersson8218 reviewed Apr 23, 2026

View reviewed changes

ryan-monroe force-pushed the export-D97667069 branch from db25b64 to 4c17fae Compare April 23, 2026 19:40

ryan-monroe force-pushed the export-D97667069 branch 2 times, most recently from b3afe9f to b9418dd Compare April 29, 2026 15:22

ryan-monroe force-pushed the export-D97667069 branch from b9418dd to 865d272 Compare April 29, 2026 18:12

ryan-monroe force-pushed the export-D97667069 branch 2 times, most recently from 4580303 to 0aaef0d Compare May 6, 2026 21:59

ryan-monroe force-pushed the export-D97667069 branch from 0aaef0d to 90ce9f6 Compare May 7, 2026 04:15

ryan-monroe force-pushed the export-D97667069 branch from 90ce9f6 to 169364d Compare May 7, 2026 15:42

ryan-monroe force-pushed the export-D97667069 branch from 169364d to 5cc8286 Compare May 8, 2026 16:47

		return first_start, expected_start


		class FuseConcatPass(ArmPass):

Conversation

ryan-monroe commented Apr 11, 2026 • edited by meta-codesync Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Empirical impact across the production EMG model fleet

Two findings worth highlighting

Net effect

Uh oh!

pytorch-bot Bot commented Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18827

❌ 3 New Failures

Uh oh!

meta-codesync Bot commented Apr 11, 2026

Uh oh!

github-actions Bot commented Apr 11, 2026

This PR needs a release notes: label

Uh oh!

ryan-monroe commented Apr 14, 2026

Uh oh!

pytorch-bot Bot commented Apr 14, 2026

Uh oh!

pytorch-bot Bot commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ninja91 Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

gggekov Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

gggekov Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

gggekov commented Apr 22, 2026

Uh oh!

oscarandersson8218 Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

Ninja91 Apr 27, 2026

Choose a reason for hiding this comment

Uh oh!

oscarandersson8218 Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ryan-monroe commented Apr 11, 2026 •

edited by meta-codesync Bot

Loading

pytorch-bot Bot commented Apr 11, 2026 •

edited

Loading

This PR needs a `release notes:` label

pytorch-bot Bot commented Apr 17, 2026 •

edited

Loading