Skip to content

Commit 363be8c

Browse files
Support torch.empty_like when lowering to Edge, and speed up GroupBasedPartitioner (#24247)
Fixes found while lowering the SmolVLA robot policy to ExecuTorch. Commits 5 and 6 change only the partitioner and do not need commits 1 to 4. **1. Accept symbolic sizes in `_empty_dim_order`.** The op's schema took `int[] size`. So `torch.empty(x.shape[0], 3)` fixed that dim to its export size, and the program failed at any other size. `torch.empty(n.item(), 3)` failed to lower at all. It now takes `SymInt[] size`. **2. Give inserted dim order copies their own memory.** A copy that `LegalizePortableDimOrderPass` inserted after a clone listed in `reinplace_extra_ops` reused the memory it read from, so values came out wrong. **3. Convert the source of a portable copy to the layout of its destination.** On main, both copies below give wrong values without an error. The portable `copy.out` kernel ignores the layout of the source, and treats the destination as contiguous when it broadcasts: ```python buffer = torch.empty(2, 3, 4, 6, memory_format=torch.channels_last) buffer.copy_(x) # x is a contiguous (2, 3, 4, 6) tensor buffer.copy_(y) # y is (3, 4, 6), broadcast to every batch ``` The in-place `copy_` used for buffer write-backs fails at runtime when the layouts differ. The pass now converts the source to the layout of the destination, and leaves alone the copies the kernel gets right. Only contiguous and 4-D channels last destinations are handled, because exir can convert a tensor only to these two layouts. **4. Support `torch.empty_like` when lowering to Edge.** Core ATen decomposition turns it into `aten.empty_permuted`, which is not core ATen, so `to_edge` fails. It cannot be decomposed right away, because in Edge its input can have another layout than in eager: `x.permute(0, 3, 1, 2)` of a contiguous tensor is channels last in eager, but a contiguous copy in Edge. So a new `ReplaceEmptyLikePass` lowers it after views have become copies. Limits: - Model inputs that are neither contiguous nor 4-D channels last still fail `to_edge`. - When `x` is the output of `avg_pool2d` on a channels last tensor, `to_executorch` later makes `x` contiguous, but `y = torch.empty_like(x)` stays channels last, so `y + x` fails at runtime. - If a partitioner preserves `empty_like`, every default-format `empty_like` is kept, even those its filter rejects, and those fail to load for lack of a kernel. **5. Make `GroupBasedPartitioner` cycle checks fast on large graphs.** It now uses bitmasks instead of per-node sets of all upstream and downstream nodes. With `XnnpackPartitioner` on SmolVLA, partitioning went from 123 min to 31 s, and its peak memory from about 30 GB to about 12 GB. **6. Remove four structures that `GroupBasedPartitioner` fills but never reads.** For the same graph, neither partitioner change moves a node to another partition (checked on thousands of random graphs). ## What was tested On Linux x86_64 CPU: 12 of the 18 new tests fail on main, and the other 6 check existing behavior. The exir tests, including exir/backend, and the XNNPACK tests have no new failures. Co-authored-by: Anthony Shoumikhin <anthony@shoumikh.in>
1 parent 4db778e commit 363be8c

14 files changed

Lines changed: 1037 additions & 171 deletions

‎exir/backend/canonical_partitioners/group_partitioner.py‎

Lines changed: 80 additions & 151 deletions
Original file line numberDiff line numberDiff line change
@@ -15,27 +15,60 @@
1515
logger.setLevel(logging.WARNING)
1616

1717

18-
class _DependencyViewer:
19-
def __init__(self, graph_module: GraphModule):
20-
self.downstreams = collections.defaultdict(set)
21-
self.upstreams = collections.defaultdict(set)
18+
def _reachable(nodes, next_nodes):
19+
"""The nodes and every node reachable from them through next_nodes."""
20+
reached = set(nodes)
21+
stack = list(reached)
22+
while stack:
23+
for node in next_nodes(stack.pop()):
24+
if node not in reached:
25+
reached.add(node)
26+
stack.append(node)
27+
return reached
2228

23-
for node in reversed(graph_module.graph.nodes):
24-
for output_node in node.users:
25-
# add output_node and output_node's downstream dependency
26-
self.downstreams[node].add(output_node)
27-
self.downstreams[node].update(self.downstreams[output_node])
2829

29-
for node in graph_module.graph.nodes:
30-
for input_node in node.all_input_nodes:
31-
self.upstreams[node].add(input_node)
32-
self.upstreams[node].update(self.upstreams[input_node])
30+
class _DependencyViewer:
31+
"""Transitive users and inputs of nodes, as bitmasks over node positions.
3332
34-
def downstreams_of(self, node: Node) -> set[Node]:
35-
return self.downstreams[node]
33+
A whole partition can then be checked for cycles with a few integer
34+
operations. Nodes must be in topological order, as FX graphs keep them.
35+
"""
3636

37-
def upstreams_of(self, node: Node) -> set[Node]:
38-
return self.upstreams[node]
37+
def __init__(self, graph_module: GraphModule, nodes):
38+
all_nodes = list(graph_module.graph.nodes)
39+
self.positions = {node: i for i, node in enumerate(all_nodes)}
40+
self.downstreams = {}
41+
self.upstreams = {}
42+
# Only these can be on a path between two of the nodes. A mask costs
43+
# one bit per position up to its highest node, so skipping the others
44+
# saves memory where most nodes are far from every partition.
45+
downstream_nodes = _reachable(nodes, lambda node: node.users)
46+
upstream_nodes = _reachable(nodes, lambda node: node.all_input_nodes)
47+
48+
for node in reversed(all_nodes):
49+
if node in downstream_nodes:
50+
mask = 0
51+
for user in node.users:
52+
mask |= (1 << self.positions[user]) | self.downstreams[user]
53+
self.downstreams[node] = mask
54+
55+
for node in all_nodes:
56+
if node in upstream_nodes:
57+
mask = 0
58+
for input_node in node.all_input_nodes:
59+
mask |= (1 << self.positions[input_node]) | self.upstreams[
60+
input_node
61+
]
62+
self.upstreams[node] = mask
63+
64+
def masks_of(self, nodes):
65+
"""Bitmasks of the nodes and of all nodes downstream and upstream of them."""
66+
node_mask = downstreams = upstreams = 0
67+
for node in nodes:
68+
node_mask |= 1 << self.positions[node]
69+
downstreams |= self.downstreams[node]
70+
upstreams |= self.upstreams[node]
71+
return node_mask, downstreams, upstreams
3972

4073

4174
class GroupBasedPartitioner(CapabilityBasedPartitioner):
@@ -80,7 +113,6 @@ def __init__(
80113
non_compute_ops=non_compute_ops,
81114
allowed_single_node_partition_ops=allowed_single_node_partition_ops,
82115
)
83-
self.dependency_viewer = _DependencyViewer(graph_module)
84116
self.node_groups = (
85117
[set(node_group) for node_group in node_groups] if node_groups else None
86118
)
@@ -95,54 +127,18 @@ def __init__(
95127
self.node_to_group[node] = i
96128
self.all_nodes_in_groups.add(node)
97129

98-
def _can_merge_partitions(self, p1, p2, partitions_by_id):
130+
def _can_merge_partitions(self, masks1, masks2):
99131
"""Check if merging two partitions would create a cycle."""
100-
from torch.fx.passes.utils.fuser_utils import validate_partition
101-
102-
p1_nodes = set(partitions_by_id[p1].nodes.keys())
103-
p2_nodes = set(partitions_by_id[p2].nodes.keys())
104-
combined_nodes = p1_nodes.union(p2_nodes)
105-
106-
# Check external users from BOTH partitions. The original code only
107-
# checked p2 under the assumption that p2 is always topologically
108-
# before p1. However, when partition groups contain nodes that span
109-
# wide topological ranges (e.g. due to shared dynamic-quantization
110-
# choose_qparams nodes), the two partitions can *interleave* in
111-
# topological order, making the single-direction check insufficient.
112-
#
113-
# We still only need to collect the *direct* external users (not
114-
# transitive ones), because dependency_viewer.downstreams_of already
115-
# returns the full transitive closure.
116-
user_nodes = []
117-
for node in combined_nodes:
118-
for user in node.users:
119-
if user not in combined_nodes:
120-
user_nodes.append(user)
121-
122-
for external_node in user_nodes:
123-
# Check if any external downstream nodes have downstream nodes in the combined partition
124-
downstream_nodes = self.dependency_viewer.downstreams_of(external_node)
125-
if any(n in combined_nodes for n in downstream_nodes):
126-
return False
127-
128-
# Final safety net: validate_partition performs a direct BFS on the
129-
# live graph edges, catching any cycle the pre-computed
130-
# dependency_viewer might miss (e.g. when the graph was transformed
131-
# after the viewer was built).
132-
if not validate_partition(list(combined_nodes)):
133-
return False
134-
135-
return True
132+
nodes, downstreams, upstreams = (m1 | m2 for m1, m2 in zip(masks1, masks2))
133+
# Fused on its own, the merged partition has a cycle exactly when a node
134+
# outside it is downstream of it and also upstream of it.
135+
return not (downstreams & upstreams & ~nodes)
136136

137137
def _process_all_nodes(
138138
self,
139139
new_partition_id,
140140
partitions_by_id,
141141
assignment,
142-
nodes_order,
143-
partitions_order,
144-
partition_users,
145-
partition_map,
146142
):
147143
"""Process nodes into a partition."""
148144
for node in reversed(self.graph_module.graph.nodes):
@@ -157,58 +153,39 @@ def _process_all_nodes(
157153
partition_id = next(new_partition_id)
158154
partition = Partition(id=partition_id, nodes=set())
159155
partitions_by_id[partition_id] = partition
160-
partitions_order[partition_id] = partition_id
161156

162157
# Add all supported nodes from the group to the partition
163158
for node in group:
164159
if self._is_node_supported(node):
165160
partition.add_node(node)
166161
assignment[node] = partition_id
167-
nodes_order[node] = partition_id
168-
169-
# Set partition users
170-
partition_users[partition_id] = {
171-
user
172-
for node in partition.nodes
173-
for user in node.users
174-
if user not in partition.nodes
175-
}
176-
177-
# Update partition map
178-
for node in partition.nodes:
179-
for user in node.users:
180-
target_id = assignment.get(user, None)
181-
if target_id is not None and target_id != partition_id:
182-
partition_map[partition_id].add(target_id)
183-
partition_map[partition_id].update(partition_map[target_id])
184162
else:
185163
partition_id = next(new_partition_id)
186-
nodes_order[node] = partition_id
187-
partitions_order[partition_id] = partition_id
188164
partitions_by_id[partition_id] = Partition(
189165
id=partition_id, nodes=[node]
190166
)
191167
assignment[node] = partition_id
192-
partition_users[partition_id] = set(node.users)
193-
194-
# Update partition map
195-
for user in node.users:
196-
target_id = assignment.get(user)
197-
if target_id is not None:
198-
partition_map[partition_id].add(target_id)
199-
partition_map[partition_id].update(partition_map[target_id])
200168

201-
def _merge_partitions(
202-
self,
203-
partitions_by_id,
204-
assignment,
205-
partition_users,
206-
partition_map,
207-
partitions_order,
208-
):
169+
def _merge_partitions(self, partitions_by_id, assignment):
209170
"""Merge partitions when possible."""
210171
# Get current partition IDs
211172
partition_ids = list(partitions_by_id.keys())
173+
# Nothing to merge, so do not spend memory on the masks.
174+
if len(partition_ids) < 2:
175+
return
176+
177+
dependency_viewer = _DependencyViewer(
178+
self.graph_module,
179+
[
180+
node
181+
for partition in partitions_by_id.values()
182+
for node in partition.nodes
183+
],
184+
)
185+
partition_masks = {
186+
partition_id: dependency_viewer.masks_of(partition.nodes)
187+
for partition_id, partition in partitions_by_id.items()
188+
}
212189

213190
# Set to track removed partitions from initial static list so we can skip them
214191
already_merged = set()
@@ -224,53 +201,25 @@ def _merge_partitions(
224201
continue
225202

226203
# Try to merge partitions if it doesn't create cycles
227-
if self._can_merge_partitions(p1, p2, partitions_by_id):
228-
self._perform_partition_merge(
229-
p1,
230-
p2,
231-
partitions_by_id,
232-
assignment,
233-
partition_users,
234-
partition_map,
235-
partitions_order,
204+
if self._can_merge_partitions(partition_masks[p1], partition_masks[p2]):
205+
self._perform_partition_merge(p1, p2, partitions_by_id, assignment)
206+
partition_masks[p1] = tuple(
207+
m1 | m2
208+
for m1, m2 in zip(partition_masks[p1], partition_masks.pop(p2))
236209
)
237210

238211
# Mark p2 as merged
239212
already_merged.add(p2)
240213

241-
def _perform_partition_merge(
242-
self,
243-
p1,
244-
p2,
245-
partitions_by_id,
246-
assignment,
247-
partition_users,
248-
partition_map,
249-
partitions_order,
250-
):
214+
def _perform_partition_merge(self, p1, p2, partitions_by_id, assignment):
251215
"""Merge partition p2 into p1."""
252216
# Merge p2 into p1
253217
partitions_by_id[p1].nodes.update(partitions_by_id[p2].nodes)
254218
for node in partitions_by_id[p2].nodes:
255219
assignment[node] = p1
256220

257-
# Update partition users
258-
all_users = partition_users[p1] | partition_users[p2]
259-
all_users.difference_update(partitions_by_id[p1].nodes)
260-
partition_users[p1] = all_users
261-
262-
# Update partition map
263-
partition_map[p1].update(partition_map[p2])
264-
265-
# Update partition order
266-
partitions_order[p1] = min(partitions_order[p1], partitions_order[p2])
267-
268221
# Remove p2
269222
del partitions_by_id[p2]
270-
del partition_users[p2]
271-
del partitions_order[p2]
272-
if p2 in partition_map:
273-
del partition_map[p2]
274223

275224
def _process_getitem_nodes(self, partitions_by_id, assignment):
276225
"""Post-process getitem nodes."""
@@ -342,35 +291,15 @@ def propose_partitions(self) -> list[Partition]:
342291
A list of proposed partitions.
343292
"""
344293
# Initialize data structures
345-
partition_map = collections.defaultdict(
346-
set
347-
) # Maps partition IDs to reachable partition IDs
348294
assignment = {} # Maps nodes to partition IDs
349295
partitions_by_id = {} # Maps partition IDs to partitions
350-
nodes_order = {} # Maps nodes to topological order
351-
partitions_order = {} # Maps partition IDs to minimum topological order
352-
partition_users = {} # Maps partition IDs to partition users
353296
new_partition_id = itertools.count()
354297

355298
# Process all nodes into partitions
356-
self._process_all_nodes(
357-
new_partition_id,
358-
partitions_by_id,
359-
assignment,
360-
nodes_order,
361-
partitions_order,
362-
partition_users,
363-
partition_map,
364-
)
299+
self._process_all_nodes(new_partition_id, partitions_by_id, assignment)
365300

366301
# Merge partitions when possible
367-
self._merge_partitions(
368-
partitions_by_id,
369-
assignment,
370-
partition_users,
371-
partition_map,
372-
partitions_order,
373-
)
302+
self._merge_partitions(partitions_by_id, assignment)
374303

375304
# Post-process getitem nodes
376305
self._process_getitem_nodes(partitions_by_id, assignment)

0 commit comments

Comments
 (0)