Skip to content

Commit 054de1f

Browse files
authored
Fix crashes, wrong outputs and lowering failures in several backends and in exir (#24222)
Fixes found while exporting models to ExecuTorch backends. ## Fixes - **exir:** `constant_prop_pass` keeps tensors that in-place ops write, also through views or in `no_grad`/`autocast` blocks. `to_backend` keeps the output node last. `to_edge` lowers `torch._check_tensor_all` and `torch.sym_ite`. `GroupBasedPartitioner` no longer makes partitions that depend on each other, unless caller node groups already do. Tensor constants get static shapes; deduplicated constants keep their data. - **AOTI (AOTInductor) and CUDA:** the missing-kernel check catches scatter and index_put fallbacks. `detach` lowers, also inside `torch.cond`. Strided outputs are copied. Transposed convolutions (1-D, 2-D, no dilation) and `torch._check_tensor_all` lower. - **XNNPACK:** the quantizer skips bfloat16 tensors, and `cat` respects scoped configs. ELU is delegated when XNNPACK can run it. - **Qualcomm:** autocast regions are inlined correctly. Unquantized bfloat16 nodes, data-dependent shapes and HTP partitions without runtime inputs stay on the CPU. Fixes for stacks, 0-d selects, RMSNorm and int64 constants. - **OpenVINO:** model data stays alive while OpenVINO 2026.4 reads it. Partitions are converted with their input types. - **Core ML:** bfloat16 ops, and ops above rank 5 outside static layout chains, stay on the CPU. - **MLX, Arm, Cortex-M, Vulkan, NXP, portable:** dtype checks for MLX tensor lists, one constant slot per weight, slice updates on a constant base; TOSA keeps unused delegate inputs for VGF; `MatmulToBmmPass` takes a constant operand; Vulkan constants above 4 dims; non-strict programs in the NXP quantizer; `cummin.out`. ## Behavior changes - A CUDA `.pte` loads only the weights it was exported with. A `.ptd` with other weights, for example after fine-tuning, needs a newly exported `.pte`. - CUDA graphs turn off when outputs do not match the ETensor layout. If only the capture call sees such a layout, that call returns an error. - `GroupBasedPartitioner` is about 2x slower on large graphs, and some graphs get a different number of partitions. `to_backend` keeps the original order of top-level nodes. - XNNPACK: weight-only quantized linears, and 8da4w or 8da8w linears (8-bit dynamic activations, 4-bit or 8-bit weights) lowered with `to_backend`, fail at export instead of giving wrong results. Delegated ELU flushes subnormal values to zero. Cats outside a scoped quantizer config are no longer quantized. - Ethos-U raises an error for a delegate with unused inputs instead of returning wrong values. We would like to cherry-pick this PR to release/1.6 after it merges. It needs #23915 there first. ## Testing Each fix has a test that fails without it. Run on an A100 (CUDA, Vulkan), an x86 CPU (OpenVINO, QNN SDK), macOS (Core ML, MLX), the TOSA reference model and VGF emulation (Arm), and the eIQ Neutron SDK (NXP). Not run: Metal, ROCm, Windows CUDA.
1 parent 403800d commit 054de1f

85 files changed

Lines changed: 2911 additions & 210 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎backends/aoti/aoti_backend.py‎

Lines changed: 39 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
import contextlib
88
import hashlib
99
import os
10+
import re
1011
import typing
1112
from abc import ABC, abstractmethod
1213
from enum import Enum
@@ -168,7 +169,8 @@ def release_moved_tensors(
168169
def collect_unsupported_fallback_kernels(cls, missing_fallback_kernels: Set[str]):
169170
"""
170171
Context manager to collect unsupported fallback kernels during compilation.
171-
Monitors both extern kernel calls and runtime lookup.
172+
Monitors extern kernel calls, runtime lookup, and the scatter and
173+
index_put fallbacks, which call their kernels directly.
172174
"""
173175
supported_kernels = cls.get_supported_fallback_kernels()
174176

@@ -178,6 +180,10 @@ def collect_unsupported_fallback_kernels(cls, missing_fallback_kernels: Set[str]
178180
original_generate_fallback_kernel_with_runtime_lookup_aot = (
179181
CppWrapperCpu.generate_fallback_kernel_with_runtime_lookup_aot
180182
)
183+
original_generate_scatter_fallback = CppWrapperCpu._generate_scatter_fallback
184+
original_generate_index_put_fallback = (
185+
CppWrapperCpu._generate_index_put_fallback
186+
)
181187

182188
def generate_c_shim_extern_kernel_call_and_collect_unsupported_kernels(
183189
self, kernel: str, *args: Any, **kwargs: Any
@@ -200,10 +206,36 @@ def generate_fallback_kernel_with_runtime_lookup_aot_and_collect_unsupported_ker
200206
self, op_overload, *args, **kwargs
201207
)
202208

209+
def collect_kernels_it_writes(original: Any) -> Any:
210+
# These fallbacks write their shim call directly, so read the
211+
# kernel names from the lines they write.
212+
def generate_and_collect_unsupported_kernels(
213+
self, *args: Any, **kwargs: Any
214+
) -> None:
215+
lines: List[str] = []
216+
writeline = self.writeline
217+
self.writeline = lambda line: (lines.append(str(line)), writeline(line))
218+
try:
219+
return original(self, *args, **kwargs)
220+
finally:
221+
self.writeline = writeline
222+
for line in lines:
223+
for kernel in re.findall(r"\b(aoti_torch_\w+)\(", line):
224+
if kernel not in supported_kernels:
225+
missing_fallback_kernels.add(kernel)
226+
227+
return generate_and_collect_unsupported_kernels
228+
203229
CppWrapperCpu.generate_c_shim_extern_kernel_call = (
204230
generate_c_shim_extern_kernel_call_and_collect_unsupported_kernels
205231
)
206232
CppWrapperCpu.generate_fallback_kernel_with_runtime_lookup_aot = generate_fallback_kernel_with_runtime_lookup_aot_and_collect_unsupported_kernels
233+
CppWrapperCpu._generate_scatter_fallback = collect_kernels_it_writes(
234+
original_generate_scatter_fallback
235+
)
236+
CppWrapperCpu._generate_index_put_fallback = collect_kernels_it_writes(
237+
original_generate_index_put_fallback
238+
)
207239

208240
try:
209241
yield
@@ -214,6 +246,12 @@ def generate_fallback_kernel_with_runtime_lookup_aot_and_collect_unsupported_ker
214246
CppWrapperCpu.generate_fallback_kernel_with_runtime_lookup_aot = (
215247
original_generate_fallback_kernel_with_runtime_lookup_aot
216248
)
249+
CppWrapperCpu._generate_scatter_fallback = (
250+
original_generate_scatter_fallback
251+
)
252+
CppWrapperCpu._generate_index_put_fallback = (
253+
original_generate_index_put_fallback
254+
)
217255

218256
@classmethod
219257
def preprocess(

‎backends/aoti/passes/replace_view_copy_with_view.py‎

Lines changed: 22 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,12 @@
3131
# natively by inductor, so map it here like the other view-copy ops.
3232
torch.ops.aten.split_copy.Tensor: torch.ops.aten.split.Tensor,
3333
ops.edge.aten.split_copy.Tensor: ops.edge.aten.split.Tensor,
34+
# Edge turns ``detach`` into ``detach_copy``, which has no c-shim either.
35+
# Without autograd it is ``alias_copy``, which Inductor can decompose,
36+
# unlike ``detach_copy``. A view would let two outputs share one handle,
37+
# and ``clone`` fails Edge verification in dim order mode.
38+
torch.ops.aten.detach_copy.default: torch.ops.aten.alias_copy.default,
39+
ops.edge.aten.detach_copy.default: ops.edge.aten.alias_copy.default,
3440
}
3541

3642

@@ -40,19 +46,26 @@ class ReplaceViewCopyWithViewPass(ExportPass):
4046
def call(self, graph_module: fx.GraphModule) -> PassResult:
4147
graph_changed = False
4248

43-
for node in graph_module.graph.nodes:
44-
if node.op != "call_function" or node.target not in _VIEW_TARGETS:
49+
# Control flow branches, such as those of torch.cond, are submodules.
50+
for module in graph_module.modules():
51+
if not isinstance(module, fx.GraphModule):
4552
continue
4653

47-
if self._has_blocking_user(node, node.users.keys()):
48-
continue
54+
module_changed = False
55+
for node in module.graph.nodes:
56+
if node.op != "call_function" or node.target not in _VIEW_TARGETS:
57+
continue
58+
59+
if self._has_blocking_user(node, node.users.keys()):
60+
continue
4961

50-
node.target = _VIEW_TARGETS[node.target]
51-
graph_changed = True
62+
node.target = _VIEW_TARGETS[node.target]
63+
module_changed = True
5264

53-
if graph_changed:
54-
graph_module.graph.lint()
55-
graph_module.recompile()
65+
if module_changed:
66+
module.graph.lint()
67+
module.recompile()
68+
graph_changed = True
5669

5770
return PassResult(graph_module, graph_changed)
5871

‎backends/apple/coreml/partition/coreml_partitioner.py‎

Lines changed: 102 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,65 @@
6363
)
6464

6565

66+
# Ops that coremltools can carry through a tensor above rank 5, by rewriting a
67+
# reshape -> transpose -> reshape chain into rank 4 steps.
68+
_HIGH_RANK_LAYOUT_TARGETS = frozenset(
69+
[
70+
exir_ops.edge.aten.view_copy.default,
71+
exir_ops.edge.aten.permute_copy.default,
72+
exir_ops.edge.aten.clone.default,
73+
exir_ops.edge.dim_order_ops._clone_dim_order.default,
74+
]
75+
)
76+
77+
78+
def _produces_tensor_above_rank_5(node: torch.fx.Node) -> bool:
79+
val = node.meta.get("val")
80+
vals = list(val) if isinstance(val, (list, tuple)) else [val]
81+
return any(isinstance(v, torch.Tensor) and v.dim() > 5 for v in vals)
82+
83+
84+
def _touches_unsupported_high_rank_tensor(
85+
node: torch.fx.Node, is_skipped: Callable[[torch.fx.Node], bool]
86+
) -> bool:
87+
"""Core ML tensors have at most rank 5.
88+
89+
coremltools 9.0 raises "Core ML only supports tensors with rank <= 5" for
90+
any tensor above rank 5 that it cannot rewrite away. It only rewrites a
91+
static, unbranched layout chain, so such a tensor is fine only when every
92+
node connected to it through tensors above rank 5 is a delegated view,
93+
permute or clone, and every node producing such a tensor has a static shape
94+
and a single user.
95+
"""
96+
seen, todo = set(), [node]
97+
while todo:
98+
n = todo.pop()
99+
if n in seen:
100+
continue
101+
seen.add(n)
102+
high_rank_inputs = [
103+
arg for arg in n.all_input_nodes if _produces_tensor_above_rank_5(arg)
104+
]
105+
high_rank_output = _produces_tensor_above_rank_5(n)
106+
if not high_rank_inputs and not high_rank_output:
107+
continue
108+
if (
109+
n.op != "call_function"
110+
or n.target not in _HIGH_RANK_LAYOUT_TARGETS
111+
or is_skipped(n)
112+
):
113+
return True
114+
if high_rank_output and (
115+
len(n.users) != 1
116+
or any(isinstance(d, torch.SymInt) for d in n.meta["val"].shape)
117+
):
118+
return True
119+
todo += high_rank_inputs
120+
if high_rank_output:
121+
todo += list(n.users)
122+
return False
123+
124+
66125
_ARG_MIN_MAX_TARGETS = (
67126
torch.ops.aten.argmax.default,
68127
torch.ops.aten.argmin.default,
@@ -83,6 +142,18 @@ def _is_arg_min_max_over_flattened_input(node: torch.fx.Node) -> bool:
83142
return dim is None
84143

85144

145+
def _reads_or_produces_bfloat16(node: torch.fx.Node) -> bool:
146+
"""coremltools 9.0 has no bfloat16 type.
147+
148+
It raises ``KeyError: torch.bfloat16`` on any op that reads or produces a
149+
bfloat16 tensor, instead of reporting the op as unsupported.
150+
"""
151+
val = node.meta.get("val")
152+
vals = list(val) if isinstance(val, (list, tuple)) else [val]
153+
vals += [arg.meta.get("val") for arg in node.all_input_nodes]
154+
return any(isinstance(v, torch.Tensor) and v.dtype == torch.bfloat16 for v in vals)
155+
156+
86157
def _is_view_op(op: torch._ops.OpOverload) -> bool:
87158
schema = op._schema
88159
if len(schema.arguments) == 0:
@@ -152,13 +223,32 @@ def should_override_support(self, node) -> bool:
152223
)
153224
return True
154225

226+
if _touches_unsupported_high_rank_tensor(
227+
node,
228+
lambda n: self.should_skip_op_for_delegation(
229+
getattr(n.target, "__name__", "").lower()
230+
),
231+
):
232+
self.log_once(
233+
f"{node.target} on a tensor above rank 5 is not supported by "
234+
"CoreML. Overriding op support."
235+
)
236+
return True
237+
155238
if _is_arg_min_max_over_flattened_input(node):
156239
self.log_once(
157240
"torch.ops.aten.{argmax, argmin}.default with dim=None is "
158241
"not supported by CoreML. Overriding op support."
159242
)
160243
return True
161244

245+
if _reads_or_produces_bfloat16(node):
246+
self.log_once(
247+
f"{node.target} on a bfloat16 tensor is not supported by CoreML. "
248+
"Overriding op support."
249+
)
250+
return True
251+
162252
# TODO: enable this after bugs in ExecuTorch's partitioner are fixed
163253
# # If lower_full_graph=False, do not partition nodes with symbolic args because it can result in symbolic args
164254
# # in the placeholders due to partitioning, which CoreML does not support
@@ -362,21 +452,26 @@ def ops_to_not_decompose(
362452
torch.ops.aten.repeat_interleave.self_int,
363453
torch.ops.aten.repeat_interleave.self_Tensor,
364454
]
455+
rejected = set()
365456
for node in ep.graph.nodes:
366457
if node.op == "call_function" and isinstance(
367458
node.target, torch._ops.OpOverload
368459
):
369460
try:
370-
if (
371-
op_support.is_node_supported(None, node)
372-
and node.target not in do_not_decompose_blocklist
373-
and not _is_view_op(node.target)
374-
):
375-
do_not_decompose.append(node.target)
461+
supported = op_support.is_node_supported(None, node)
376462
except Exception as e:
377463
# CoreML's op_support.is_node_supported will sometimes throw
378464
# for unsupported ops, rather than returning False
379465
self.log_once(
380466
f"Encountered exception when checking node support, treating node as unsupported: {e}"
381467
)
382-
return do_not_decompose, None
468+
supported = False
469+
if not supported:
470+
rejected.add(node.target)
471+
elif node.target not in do_not_decompose_blocklist and not _is_view_op(
472+
node.target
473+
):
474+
do_not_decompose.append(node.target)
475+
# Without a per-node filter, a preserved op stays whole at every call, so
476+
# one rejected call would be left with no kernel.
477+
return [op for op in do_not_decompose if op not in rejected], None

‎backends/apple/coreml/test/test_coreml_partitioner.py‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -386,6 +386,61 @@ def forward(self, x):
386386
self.assertIn("executorch_call_delegate", op_names)
387387
self.assertNotIn("aten.randn.default", op_names)
388388

389+
def test_ops_above_rank_5_are_skipped(self):
390+
"""
391+
Core ML tensors have at most rank 5. coremltools only rewrites an
392+
unbranched view, permute, view chain through a higher rank (a window
393+
partition), and raises on any other op above rank 5. The partitioner
394+
must leave those ops to the portable backend and keep such a layout
395+
chain delegated.
396+
"""
397+
398+
class Model(torch.nn.Module):
399+
def forward(self, x):
400+
windows = x.view(2, 2, 2, 2, 2, 8).permute(0, 1, 3, 2, 4, 5)
401+
windows = windows.reshape(-1, 4, 8).relu()
402+
branched = x.view(2, 2, 2, 2, 2, 8).permute(0, 1, 3, 2, 4, 5)
403+
branched = (
404+
branched.reshape(-1, 4, 8).sum() + branched.reshape(32, 8).sum()
405+
)
406+
stacked = torch.stack([x.unsqueeze(0), x.unsqueeze(0) * 2])
407+
return windows.sum() + branched + stacked.sum(0).relu().sum()
408+
409+
model = Model().eval()
410+
x = torch.randn(2, 4, 4, 8)
411+
ep = torch.export.export(model, (x,), strict=True)
412+
edge = executorch.exir.to_edge_transform_and_lower(
413+
ep, partitioner=[CoreMLPartitioner()]
414+
)
415+
op_names = [
416+
n.target.__name__
417+
for n in edge.exported_program().graph.nodes
418+
if n.op == "call_function"
419+
]
420+
self.assertIn("executorch_call_delegate", op_names)
421+
self.assertIn("aten.sum.dim_IntList", op_names)
422+
self.assertEqual(op_names.count("aten.permute_copy.default"), 1)
423+
if _TEST_RUNTIME:
424+
program = Runtime.get().load_program(edge.to_executorch().buffer)
425+
out = program.load_method("forward").execute((x,))[0]
426+
self.assertTrue(torch.allclose(out, model(x), atol=5e-2, rtol=5e-2))
427+
428+
# A skipped permute leaves the chain's views with no rank 6 rewrite.
429+
edge = executorch.exir.to_edge_transform_and_lower(
430+
ep,
431+
partitioner=[
432+
CoreMLPartitioner(
433+
skip_ops_for_coreml_delegation=["aten.permute_copy.default"]
434+
)
435+
],
436+
)
437+
op_names = [
438+
n.target.__name__
439+
for n in edge.exported_program().graph.nodes
440+
if n.op == "call_function"
441+
]
442+
self.assertEqual(op_names.count("aten.permute_copy.default"), 2)
443+
389444
def test_argmax_argmin_dim_none_is_skipped(self):
390445
"""
391446
Regression test for https://github.com/pytorch/executorch/issues/11715.
@@ -433,6 +488,39 @@ def forward(self, x):
433488
self.assertIn("executorch_call_delegate", op_names)
434489
self.assertNotIn("aten.argmax.default", op_names)
435490

491+
def test_bfloat16_ops_are_skipped(self):
492+
"""
493+
coremltools has no bfloat16 type and raises on ops that use it. The
494+
partitioner must leave those ops to the portable backend and still
495+
delegate the rest, also when the same op has a float32 call that Core
496+
ML takes (as under bfloat16 autocast).
497+
"""
498+
499+
class Model(torch.nn.Module):
500+
def forward(self, q):
501+
a = torch.nn.functional.scaled_dot_product_attention(q, q, q)
502+
b = q.to(torch.bfloat16)
503+
c = torch.nn.functional.scaled_dot_product_attention(b, b, b)
504+
return a + c.float()
505+
506+
model = Model().eval()
507+
x = torch.randn(1, 2, 4, 8)
508+
ep = torch.export.export(model, (x,), strict=True)
509+
edge = executorch.exir.to_edge_transform_and_lower(
510+
ep, partitioner=[CoreMLPartitioner()]
511+
)
512+
op_names = [
513+
n.target.__name__
514+
for n in edge.exported_program().graph.nodes
515+
if n.op == "call_function"
516+
]
517+
self.assertIn("executorch_call_delegate", op_names)
518+
self.assertNotIn("aten.scaled_dot_product_attention.default", op_names)
519+
if _TEST_RUNTIME:
520+
program = Runtime.get().load_program(edge.to_executorch().buffer)
521+
out = program.load_method("forward").execute((x,))[0]
522+
self.assertTrue(torch.allclose(out, model(x), atol=5e-2, rtol=5e-2))
523+
436524
def test_tied_embedding_is_quantized_with_its_linear(self):
437525
"""A weight that is both an embedding table and an output projection lowers.
438526

0 commit comments

Comments
 (0)