Skip to content

Commit 3eae0b0

Browse files
committed
Qualcomm AI Engine Direct - Keep positional sin and cos in float32
In float16 mode QNN runs every delegated float32 op in float16. A timestep or rotary embedding takes sin and cos of large angles, like t*freqs or positions/timescale, and in float16 these come out badly wrong, for 2 reasons. float16 cannot hold the angle: its values are 1.0 apart from 1024 to 2048, and a timestep embedding with a 4e-3 period reaches 1571. And HTP's float16 Sin and Cos lose more on top of that for large angles. On the QNN SDK's x86 simulator for SM8650, the largest error against float32 was 1.39 for timestep embedding angles of 1000 to 2000, and 0.50 for rotary embedding angles of 300 to 520 (positions up to 511). A float32 sin of the float16 angle is still off by 0.95 in the first case, so a better Sin alone would not fix it. A new to_edge pass, TagPositionalSinCos, tags such a float32 sin or cos for CPU fallback, together with the ops that compute its angle, so they compute in float32 as the model does. The partitioner already leaves nodes tagged QCOM_FALLBACK_NODE on CPU. The pass acts for HTP in float16 precision only, and needs the compiler specs, which to_edge_transform_and_lower_to_qnn, the QNN recipe and the QNN tester give it. The HTP pass manager orders it after every other default pass, last before ResolveDebugHandle, so it sees the graph the partitioner sees. It is on by default, logs a warning when it tags ops, and can be turned off in passes_job, for example to keep the delegated graph of an older program, whose runtime may not have the CPU kernels these ops need: passes_job = get_qnn_pass_manager_cls().get_capture_program_passes() passes_job[TagPositionalSinCos][QCOM_PASS_ACTIVATE_KEY] = False to_edge_transform_and_lower_to_qnn( module, inputs, specs, passes_job=passes_job ) The QNN FP16 recipe takes the same passes_job, and rejects a partial or malformed one with a ValueError instead of failing inside lowering. It applies only when the angle is built from graph inputs, constants, integer positions and sizes (like position ids or x.shape[1]) by cheap elementwise and shape ops: arithmetic, pow, exp, log, casts, views, slices, expand, permute, cat, and the arange and where that a linspace becomes. The sin or cos must also be larger than each graph input and integer tensor its float ops read, as an angle that spreads a few positions or timesteps over a frequency axis is. Float parameters and buffers do not count, so a batch of one timestep still qualifies, while sin(x * 30) of an image, float or integer, stays delegated. Integer positions built only from graph inputs and constants, like attention_mask.cumsum(-1), mask.sum(-1) + ones(n).cumsum(0) or arange(n) < k, move to CPU with the angle, including their casts and comparisons. This matters because I64toI32 puts an int32 cast after each int64 op, and QNN gives that op the int32 output: a cast left on CPU behind a delegated int64 op would read the wrong data. Integers computed from activations stay delegated, and so does any angle built from them, so every tagged op reads only graph inputs and constants and no delegate is split. A sin or cos of an activation, for example after a linear or a conv, stays delegated, so no heavy op moves to CPU. So does a sin or cos of a graph input that the graph only moves or casts: the graph does not compute that angle, so the caller can keep it small. Quantized nodes, the quantized mode and other backends are not affected. A rotary embedding that builds its angle with a matmul, as Hugging Face models do with inv_freq @ position_ids, is not covered. QNN keeps matmul whole for HTP, and a whole matmul has no CPU kernel, so it cannot simply stay on CPU. The check for HTP float16 is now one helper, is_htp_fp16, used by the new pass, RecomposeRmsNorm and the partitioner. Results on the simulator: - For the 2 cases above the largest error is now 0.0002, the float16 rounding of the delegated cat after them, with the same number of delegates. - SmolVLA, whose language model builds rotary angles from position ids this way: the largest error of the keys it caches drops from 0.79 to 0.19, with the same number of delegates. With a fix for its uint8 camera input cast also applied, the end to end action check drops from 0.077 to 0.033. Tests: - A delegate test lowers a sinusoidal embedding with angles up to 1571, built with linspace inside forward, and checks its output. - Another lowers rotary angles from attention_mask.cumsum(-1) positions plus an int64 constant made in forward, up to 500, and checks its output, so the integer ops and their int32 casts run on CPU end to end. - A partitioner test checks that such a sin and its angle go to CPU while the cos of a linear stays delegated, in one delegate, and that turning the pass off delegates the sin again. - Unit tests, which do not initialize the QNN backend, run the QNN to_edge passes: a linspace timestep angle, an exp(-log(10000) * i) frequency, a batch of one, an angle from an int64 cumsum, from offset positions or from a count over a mask as large as the angle, and aranges of a dynamic size or int input are tagged with every op they come from; an angle computed from a linear or from integers computed from an activation (including a float sum), sin(x * 30) of a float or integer graph input, a sin of a graph input or of its view, a float16 sin, quantized nodes, quantized HTP and the GPU backend are not. Another checks that the QNN FP16 recipe turns the pass off with the same passes_job and rejects partial or malformed ones.
1 parent 215b45f commit 3eae0b0

11 files changed

Lines changed: 591 additions & 14 deletions

File tree

‎backends/qualcomm/_passes/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@
7575
from .replace_arange_args import ReplaceArangeArgs
7676
from .resolve_debug_handle import ResolveDebugHandle
7777
from .seq_mse import SeqMSE
78+
from .tag_positional_sin_cos import TagPositionalSinCos
7879
from .tag_quant_io import TagQuantIO
7980

8081
__all__ = [
@@ -149,5 +150,6 @@
149150
ReplaceArangeArgs,
150151
ResolveDebugHandle,
151152
SeqMSE,
153+
TagPositionalSinCos,
152154
TagQuantIO,
153155
]

‎backends/qualcomm/_passes/backends/htp/qnn_htp_pass_manager.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@
88
DecomposeReciprocal,
99
RecomposeHadamard,
1010
RemoveRedundancy,
11+
ResolveDebugHandle,
12+
TagPositionalSinCos,
1113
)
1214
from executorch.backends.qualcomm._passes.qnn_pass_manager import QnnPassManager
1315

@@ -25,6 +27,7 @@ def get_default_pass_activations(cls):
2527
# The current change is intended to proactively prepare for the upcoming deprecation of the export pipeline.
2628
pass_activations = super().get_default_pass_activations()
2729
pass_activations.extend([(DecomposeReciprocal, True)])
30+
pass_activations.append((TagPositionalSinCos, True))
2831
return pass_activations
2932

3033
@classmethod
@@ -33,6 +36,13 @@ def get_passes_dependency_for_capture_program(cls):
3336
# The current change is intended to proactively prepare for the upcoming deprecation of the export pipeline.
3437
deps = super().get_passes_dependency_for_capture_program()
3538
deps.update({DecomposeReciprocal: [RemoveRedundancy]})
39+
# Last before ResolveDebugHandle, so it tags the graph the partitioner sees.
40+
deps[TagPositionalSinCos] = [
41+
p
42+
for p, _ in cls.get_default_pass_activations()
43+
if p not in (TagPositionalSinCos, ResolveDebugHandle)
44+
]
45+
deps[ResolveDebugHandle] = deps[ResolveDebugHandle] + [TagPositionalSinCos]
3646
return deps
3747

3848
@classmethod

‎backends/qualcomm/_passes/recompose_rms_norm.py‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,6 @@
1515
)
1616
from executorch.backends.qualcomm.serialization.qc_schema import (
1717
QnnExecuTorchBackendType,
18-
QnnExecuTorchHtpPrecision,
1918
)
2019
from executorch.backends.qualcomm.serialization.qc_schema_serialize import (
2120
flatbuffer_to_option,
@@ -108,6 +107,7 @@ def __init__(
108107
# Imported here, since partition depends on _passes.
109108
from executorch.backends.qualcomm.partition.utils import (
110109
generate_qnn_executorch_option,
110+
is_htp_fp16,
111111
)
112112

113113
options = flatbuffer_to_option(
@@ -116,11 +116,7 @@ def __init__(
116116
self.may_be_htp = (
117117
options.backend_type == QnnExecuTorchBackendType.kHtpBackend
118118
)
119-
self.htp_fp16 = (
120-
self.may_be_htp
121-
and options.htp_options is not None
122-
and options.htp_options.precision == QnnExecuTorchHtpPrecision.kHtpFp16
123-
)
119+
self.htp_fp16 = is_htp_fp16(options)
124120
self.rms_norm_target = exir_ops.edge.aten.rms_norm.default
125121
self.skip_targets = [
126122
exir_ops.edge.aten.to.dtype,
Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
# All rights reserved.
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
6+
import logging
7+
from typing import Dict, Set
8+
9+
import torch
10+
from executorch.backends.qualcomm.serialization.qc_schema_serialize import (
11+
flatbuffer_to_option,
12+
)
13+
from executorch.backends.qualcomm.utils.constants import (
14+
QCOM_FALLBACK_NODE,
15+
QCOM_QUANT_ATTRS,
16+
)
17+
from executorch.exir.dialects._ops import ops as exir_ops
18+
from executorch.exir.pass_base import ExportPass, PassResult
19+
from torch.fx.experimental.symbolic_shapes import statically_known_true
20+
21+
logger = logging.getLogger(__name__)
22+
23+
_SIN_COS_OPS = {exir_ops.edge.aten.sin.default, exir_ops.edge.aten.cos.default}
24+
25+
# Ops that only move or cast values. An angle that is a graph input moved this
26+
# way alone is not computed by the graph, so the caller can keep it small.
27+
_VIEW_OPS = {
28+
exir_ops.edge.aten._to_copy.default,
29+
exir_ops.edge.aten.cat.default,
30+
exir_ops.edge.aten.clone.default,
31+
exir_ops.edge.aten.expand_copy.default,
32+
exir_ops.edge.aten.permute_copy.default,
33+
exir_ops.edge.aten.select_copy.int,
34+
exir_ops.edge.aten.slice_copy.Tensor,
35+
exir_ops.edge.aten.squeeze_copy.dims,
36+
exir_ops.edge.aten.unsqueeze_copy.default,
37+
exir_ops.edge.aten.view_copy.default,
38+
exir_ops.edge.dim_order_ops._to_dim_order_copy.default,
39+
}
40+
41+
# The cheap elementwise and shape ops a positional angle, like t * freqs,
42+
# positions / 10000 ** (2 * arange(n) / d) or t * exp(-log(10000) * arange(n)
43+
# / n), is built from. where comes from a decomposed linspace.
44+
_ANGLE_OPS = _VIEW_OPS | {
45+
exir_ops.edge.aten.add.Tensor,
46+
exir_ops.edge.aten.arange.start_step,
47+
exir_ops.edge.aten.div.Tensor,
48+
exir_ops.edge.aten.exp.default,
49+
exir_ops.edge.aten.log.default,
50+
exir_ops.edge.aten.mul.Tensor,
51+
exir_ops.edge.aten.neg.default,
52+
exir_ops.edge.aten.pow.Tensor_Tensor,
53+
exir_ops.edge.aten.reciprocal.default,
54+
exir_ops.edge.aten.sub.Tensor,
55+
exir_ops.edge.aten.where.self,
56+
}
57+
58+
# Positions are also built from masks, like attention_mask.cumsum(-1),
59+
# mask.sum(-1) + ones(n).cumsum(0) or arange(n) < k.
60+
_POSITION_OPS = _ANGLE_OPS | {
61+
exir_ops.edge.aten.cumsum.default,
62+
exir_ops.edge.aten.full.default,
63+
exir_ops.edge.aten.sum.dim_IntList,
64+
exir_ops.edge.aten.eq.Scalar,
65+
exir_ops.edge.aten.eq.Tensor,
66+
exir_ops.edge.aten.ge.Scalar,
67+
exir_ops.edge.aten.ge.Tensor,
68+
exir_ops.edge.aten.gt.Scalar,
69+
exir_ops.edge.aten.gt.Tensor,
70+
exir_ops.edge.aten.le.Scalar,
71+
exir_ops.edge.aten.le.Tensor,
72+
exir_ops.edge.aten.lt.Scalar,
73+
exir_ops.edge.aten.lt.Tensor,
74+
exir_ops.edge.aten.ne.Scalar,
75+
exir_ops.edge.aten.ne.Tensor,
76+
}
77+
78+
79+
def _is_exact(node: torch.fx.Node) -> bool:
80+
val = node.meta.get("val")
81+
return isinstance(val, torch.SymInt) or (
82+
isinstance(val, torch.Tensor) and not val.is_floating_point()
83+
)
84+
85+
86+
class TagPositionalSinCos(ExportPass):
87+
"""
88+
On HTP in float16 precision, tag the unquantized float32 sin and cos nodes
89+
whose angle the graph computes from its inputs, constants and integer
90+
positions with only the ops in _ANGLE_OPS, as a timestep or rotary
91+
embedding does, and that are larger than each input they read, together
92+
with those ops, so that the partitioner leaves them on CPU.
93+
94+
float16 cannot hold such an angle precisely: its values are 1.0 apart from
95+
1024 to 2048, where the angles of a timestep embedding can reach, and HTP's
96+
float16 Sin and Cos lose more on top of that for large angles. On CPU
97+
these nodes compute in float32, as the model does. A sin or cos whose angle
98+
reads an activation, even through integers, is left alone, so no conv or
99+
linear moves to CPU and the delegate is not split.
100+
101+
The tagged ops then need CPU kernels at runtime. To delegate them as
102+
before, take passes_job = get_qnn_pass_manager_cls().get_capture_program_passes()
103+
and set passes_job[TagPositionalSinCos][QCOM_PASS_ACTIVATE_KEY] = False.
104+
"""
105+
106+
def __init__(self, edge_program, compiler_specs=None):
107+
super(TagPositionalSinCos, self).__init__()
108+
self.user_inputs = set(edge_program.graph_signature.user_inputs)
109+
self.htp_fp16 = False
110+
if compiler_specs is not None:
111+
# Imported here, since partition depends on _passes.
112+
from executorch.backends.qualcomm.partition.utils import (
113+
generate_qnn_executorch_option,
114+
is_htp_fp16,
115+
)
116+
117+
self.htp_fp16 = is_htp_fp16(
118+
flatbuffer_to_option(
119+
generate_qnn_executorch_option(compiler_specs)
120+
).backend_options
121+
)
122+
123+
def call(self, graph_module: torch.fx.GraphModule):
124+
if not self.htp_fp16:
125+
return PassResult(graph_module, False)
126+
nodes, positions = set(), {}
127+
for node in graph_module.graph.nodes:
128+
if (
129+
node.target in _SIN_COS_OPS
130+
and QCOM_QUANT_ATTRS not in node.meta
131+
and node.meta["val"].dtype == torch.float32
132+
):
133+
angle_nodes = self._get_angle_nodes(node.args[0], positions)
134+
if any(
135+
n.target not in _VIEW_OPS for n in angle_nodes
136+
) and self._broadcasts_inputs(node, angle_nodes):
137+
nodes |= angle_nodes | {node}
138+
for node in nodes:
139+
node.meta[QCOM_FALLBACK_NODE] = True
140+
if nodes:
141+
logger.warning(
142+
f"TagPositionalSinCos keeps {len(nodes)} ops of positional sin "
143+
"and cos on CPU in float32, which needs their CPU kernels. "
144+
"To delegate them, set passes_job[TagPositionalSinCos]"
145+
"[QCOM_PASS_ACTIVATE_KEY] = False in the passes_job from "
146+
"get_qnn_pass_manager_cls().get_capture_program_passes()."
147+
)
148+
return PassResult(graph_module, bool(nodes))
149+
150+
def _broadcasts_inputs(
151+
self, node: torch.fx.Node, angle_nodes: Set[torch.fx.Node]
152+
) -> bool:
153+
# A positional angle spreads a few positions or timesteps over a
154+
# frequency axis, so it is larger than every user input and integer
155+
# tensor its float ops read. An angle as large as one, like an image
156+
# times a scale, stays delegated. Integer positions may read larger
157+
# masks.
158+
numel = node.meta["val"].numel()
159+
return all(
160+
statically_known_true(arg.meta["val"].numel() < numel)
161+
for angle_node in angle_nodes
162+
if not _is_exact(angle_node)
163+
for arg in angle_node.all_input_nodes
164+
if isinstance(arg.meta.get("val"), torch.Tensor)
165+
and (arg.name in self.user_inputs or _is_exact(arg))
166+
)
167+
168+
def _get_angle_nodes(
169+
self, angle: torch.fx.Node, positions: Dict[torch.fx.Node, Set[torch.fx.Node]]
170+
) -> Set[torch.fx.Node]:
171+
"""
172+
The nodes computing `angle` from graph inputs, constants and integer
173+
positions, or an empty set if another op takes part.
174+
"""
175+
nodes, stack = set(), [angle]
176+
while stack:
177+
node = stack.pop()
178+
if node in nodes or node.op != "call_function":
179+
continue
180+
if _is_exact(node):
181+
position_nodes = self._get_position_nodes(node, positions)
182+
if not position_nodes and not isinstance(
183+
node.meta["val"], torch.SymInt
184+
):
185+
return set()
186+
nodes |= position_nodes
187+
continue
188+
if node.target not in _ANGLE_OPS or QCOM_QUANT_ATTRS in node.meta:
189+
return set()
190+
nodes.add(node)
191+
stack.extend(node.all_input_nodes)
192+
return nodes
193+
194+
def _get_position_nodes(
195+
self, node: torch.fx.Node, positions: Dict[torch.fx.Node, Set[torch.fx.Node]]
196+
) -> Set[torch.fx.Node]:
197+
"""
198+
`node` and the integer nodes it is computed from, if they are built
199+
from graph inputs, constants and sizes with the ops in _POSITION_OPS
200+
only, or an empty set otherwise.
201+
202+
Integers are exact, so they only need to move with the angle when they
203+
read nothing else. Then the casts I64toI32 puts after int64 producers
204+
move together with those producers.
205+
"""
206+
if node not in positions:
207+
positions[node] = set()
208+
nodes = {node}
209+
for arg in node.all_input_nodes:
210+
if arg.op != "call_function" or isinstance(
211+
arg.meta.get("val"), torch.SymInt
212+
):
213+
continue
214+
arg_nodes = (
215+
self._get_position_nodes(arg, positions)
216+
if _is_exact(arg)
217+
else set()
218+
)
219+
if not arg_nodes:
220+
nodes = set()
221+
break
222+
nodes |= arg_nodes
223+
if node.target not in _POSITION_OPS or QCOM_QUANT_ATTRS in node.meta:
224+
nodes = set()
225+
positions[node] = nodes
226+
return positions[node]

‎backends/qualcomm/partition/qnn_partitioner.py‎

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,6 @@
2222
from executorch.backends.qualcomm.serialization.qc_schema import (
2323
HtpArch,
2424
QnnExecuTorchBackendType,
25-
QnnExecuTorchHtpPrecision,
2625
)
2726
from executorch.backends.qualcomm.serialization.qc_schema_serialize import (
2827
flatbuffer_to_option,
@@ -62,6 +61,7 @@
6261
get_skip_decomp_table,
6362
has_data_dependent_shape,
6463
has_unsupported_dtype,
64+
is_htp_fp16,
6565
warn_if_dilated_conv_may_not_fit_vtcm,
6666
)
6767

@@ -102,17 +102,12 @@ def __init__(
102102
self.phase = phase
103103
self.soc_info = python_options.soc_info
104104
self.backend_type = python_options.backend_options.backend_type
105-
htp_options = python_options.backend_options.htp_options
106105
# Gate for the VTCM diagnostic in is_node_supported. Only fp16 HTP: the
107106
# quantized path rejects an untileable dilated conv at finalize, so it
108107
# needs no warning. Not gated on log level, unlike the backend's other
109108
# diagnostics -- the failure it points at is a silent unresponsive DSP
110109
# with no other clue in the log, so it has to be visible by default.
111-
self.warn_dilated_conv_vtcm = (
112-
self.backend_type == QnnExecuTorchBackendType.kHtpBackend
113-
and htp_options is not None
114-
and htp_options.precision == QnnExecuTorchHtpPrecision.kHtpFp16
115-
)
110+
self.warn_dilated_conv_vtcm = is_htp_fp16(python_options.backend_options)
116111
self.nodes_to_wrappers = defaultdict(dict)
117112
target_socs = (
118113
[

‎backends/qualcomm/partition/utils.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,11 @@
1414
QNN_TENSOR_TYPE_MAP,
1515
)
1616

17+
from executorch.backends.qualcomm.serialization.qc_schema import (
18+
QnnExecuTorchBackendOptions,
19+
QnnExecuTorchBackendType,
20+
QnnExecuTorchHtpPrecision,
21+
)
1722
from executorch.backends.qualcomm.utils.constants import (
1823
QCOM_BYPASS_NODE,
1924
QCOM_QNN_COMPILE_SPEC,
@@ -46,6 +51,17 @@ def generate_qnn_executorch_option(
4651
return qnn_compile_spec_buffer
4752

4853

54+
def is_htp_fp16(backend_options: QnnExecuTorchBackendOptions) -> bool:
55+
"""
56+
Whether the backend options target HTP in float16 precision.
57+
"""
58+
return (
59+
backend_options.backend_type == QnnExecuTorchBackendType.kHtpBackend
60+
and backend_options.htp_options is not None
61+
and backend_options.htp_options.precision == QnnExecuTorchHtpPrecision.kHtpFp16
62+
)
63+
64+
4965
def estimate_conv_vtcm_working_set(
5066
in_channels: int,
5167
spatial: List[int],

0 commit comments

Comments
 (0)