Repository navigation
Qualcomm AI Engine Direct - Keep positional sin and cos in float32 - #24601
shoumikhin wants to merge 28 commits into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24601
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 2 PendingAs of commit 70abb65 with merge base b69ece4 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@claude /pr-review |
|
Claude finished @shoumikhin's task in 31m 8s —— View job Request Changes: Detailed reviewPR Review: #24601SummaryThe new commit adds
Scope: only Code Quality
Testing
API Design
Backward Compatibility
Performance
RecommendationRequest Changes Before merging, add a test that lowers an int64-mask cumsum angle and an int64-buffer angle through |
5ea2021 to
9179a13
Compare
|
@claude /pr-review |
|
Claude finished @shoumikhin's task in 39m 39s —— View job Request Changes: the default-on Full reviewPR Review: #24601SummaryThe PR adds
Scope: only the last commit, Code Quality
Backward Compatibility
API Design
RecommendationRequest Changes: bound the default-on tagging (or make it opt-in), and make the recipe opt-out hard to misuse. The int64-constant concern is low confidence; the suggested end-to-end test would confirm or clear it. |
|
@qti-horodnic could you take a look? It keeps positional sin and cos in float32 on HTP, so large angles stay accurate. |
9179a13 to
70a8bc6
Compare
70a8bc6 to
3eae0b0
Compare
|
Thanks. Updated:
|
|
@claude /pr-review |
|
Claude finished @shoumikhin's task in 42m 26s —— View job Request Changes: the new size bound in Detailed reviewPR Review: #24601SummaryThe PR adds The size bound added in this round compares the sin/cos against the wrong tensors:
Only the last commit ( Code Quality
Performance
RecommendationRequest Changes The size bound exists to tell positional angles apart from data angles. As written, it leaves the PR's own |
3eae0b0 to
bd435fd
Compare
|
Thanks. Changes in the new push:
|
|
@claude /pr-review |
|
Claude finished @shoumikhin's task in 47m 19s —— View job PR Review: #24601Approve: no correctness problems found in Detailed reviewSummaryThe two commits add Code Quality
Testing
API Design
Backward Compatibility
RecommendationApprove: nothing blocks merging. The four items are minor and can be handled here or in a follow-up.
|
The backend mapped torch.float64 to QNN_DATATYPE_FLOAT_32 and delegated float64 ops. The data was never converted: float64 constants and graph inputs reached QNN as raw 8-byte values read as 4-byte floats. So float64 math gave wrong results with no error. For example, (x.double() * 2 + 1).float() returned zeros on the QNN SDK's x86 simulator for SM8650 in float16 mode, and a sinusoidal timestep embedding computed in float64, as openpi and LeRobot's pi0, pi05 and SmolVLA do, was off by up to 2.0. float64 is now left out of the QNN type map, as bfloat16 already is. A node that reads or writes a float64 tensor stays on CPU, where float64 is computed exactly. Ops on other dtypes are delegated as before. The quantizer's dtype check reads the same map, so it now treats a float64 input as having no QNN type instead of failing on the missing key. The To builder's uint8 cast through int32 on HTP no longer needs a float64 case, since a float64 cast now stays on CPU. On the simulator, both examples above now match PyTorch exactly, and the error of SmolVLA's denoising step drops from 0.32 to 0.0027 (largest absolute error, outputs up to 3.2). New tests lower float64 math between float32 ops, unquantized and quantized, and check the output.
When the QNN partitioner delegates a mutable buffer, the delegate owns
the buffer's memory. The QNN runtime does not load the buffer's starting
value into that memory yet, so the buffer starts from whatever the
memory held. For an int64 counter the result is wrong with no error:
self.register_buffer("count", torch.zeros(2, dtype=torch.long))
...
self.count.add_(1)
# count comes back as [181093535, 1] instead of [1, 1]; the first
# value changes from run to run
The partitioner now leaves integer mutable buffers on CPU, and any
other buffer that is not floating point, even with the default
skip_mutable_buffer=False. The ops that update the buffer can still run
in the delegate: the buffer goes in as a delegate input, and the new
value comes back as a delegate output that CPU writes to the buffer, as
skip_mutable_buffer=True already does for every buffer. Floating-point
buffers are delegated as before, so a float counter still starts from
leftover memory.
This matters more with the next change, which casts every int64 result
to int32: then an update such as count.add_((x > 0.5).sum(-1)), which
runs on CPU today for an int64 or an int32 count, could be delegated
together with its buffer.
A test lowers int64 and int32 counters updated by a constant and an
argmax, and checks that only the float buffer is delegated.
Many QNN builders read and write integer tensors, such as indices,
as int32. I64toI32 turns int64 inputs and constants into int32, and it
added an int32 cast after the ops in a fixed list: argmax, argmin, arange, cumsum, full, scalar_tensor and
casts. Other ops make int64 too, like a sum or a product of bools or
integers, and the indices of topk and sort. Their results stayed int64,
which caused lowering errors and wrong results:
- An op that needs equal dtypes failed when the pass traced the graph
again:
(x > 0.5).sum(-1).masked_scatter(mask, k)
RuntimeError: masked_scatter: expected self and source to have
same dtypes but got torch.int64 and torch.int32
- A value that went into or out of QNN this way was read with the
wrong size, with no error. For example x[:, (x > 0.5).sum(-1)]
picked the wrong columns, and topk indices [1, 2] came back as
8589934593.
The pass now casts every int64 result to int32, also the ones read
through getitem, instead of relying on a list. The int64 casts that the
pass adds for the index of gather and scatter stay, because the CPU
kernels need int64 there. The 8-bit quantized embedding now gets the
same index cast, since its CPU kernel only takes int64. An input listed
in skip_node, like the token ids of the Llama flow, is still int64, so
it gets no index cast. Int64 graph outputs are still cast back to int64
at the end. For a multi-output op such as topk, whose input can now be
int32, each returned output is cast back through its getitem.
This also changes graphs that worked before, when they have such an
int64 result. A value that does not fit in int32 now wraps around, as
already happens to int64 inputs and to the listed ops. When the op that
makes the value stays on CPU, the new cast can become a QNN delegate
that only converts the dtype, which the next change keeps on CPU, or
split an existing delegate in 2.
The previous change keeps integer mutable buffers on CPU, so a buffer
updated from such a value still gets the right value. Graphs whose int64
results all come from the listed ops are unchanged.
The debug handle test model, in test_passes and in the rework pass
tests, gains one node, the cast of its topk indices, so the handles
after it move up by 1.
The test for integer mutable buffers also updates the counters by a sum
of bools now, an int64 result that this change casts.
… CPU
I64toI32 casts int64 graph inputs and int64 results to int32, because
QNN handles integer tensors as int32, and casts int64 graph outputs
back. When the ops around such a cast stay on CPU, the casts can end up
in a partition of their own. QNN then gets a delegate that only converts
dtypes, which costs a graph execution on every inference and does no
work of the model. For example:
(x > 0.5).sum(-1).masked_scatter(mask, k)
# sum and masked_scatter run on CPU
# before: 3 delegates, [gt], [cast k, cast sum], [cast the output back]
# after: 1 delegate, [gt]
I64toI32 now marks every cast it inserts, and the QNN partitioner keeps
a partition made only of such casts on CPU. A partition that holds a
marked cast next to other ops is unchanged, and so is a cast the model
makes itself.
A new test checks that this graph lowers to the gt delegate alone and
that a model that is only a cast is still delegated. Another checks that
only the mark counts, not the op, since other passes insert _to_copy
too. The delegate tests of Int64Results now expect 2 delegates instead
of 3.
…n I64toI32
I64toI32 casts every int64 result to int32, because QNN handles integer
tensors as int32. Some ops have no QNN builder at all, so they always
run on CPU, where int64 works. For them the cast only did harm:
x.prod(-1) # int32 input [[65536, 65536]]
# before: 0, the product wrapped around in the int32 cast
# after: 4294967296, as in eager
s, zp = choose_qparams_per_token_asymmetric(x, torch.int8)
quantize_per_token(x, s, zp, -128, 127, torch.int8)
# before: the runtime aborted, since the CPU kernel needs an int64
# zero point and got the int32 cast
# after: runs and matches eager
The int64 results of an ATen or quantized_decomposed op that QNN has no
builder for now stay int64. A QNN op that reads such a result gets an
int32 cast of it, so int64 still never enters QNN. Ops with a builder,
and ops of other namespaces such as custom ops, are cast as before, and
a CPU op reads their results through that cast, as before. A CPU op that
reads an int32 tensor together with a kept int64 result reads the int32
cast of that result too, so that an op like masked_scatter gets
matching dtypes. A value that a QNN op reads still wraps around when it
does not fit in int32.
max_pool2d_with_indices_backward runs on CPU too, and its kernel needs
int64 indices, which the pass now casts to int32 when they come from a
getitem. It gets the same int64 index cast as gather and scatter.
A test checks that a product keeps its int64 value, that the CPU
quantize_per_token reads an int64 zero point, that an index op reads a
product as int32, that masked_scatter of a product into an int64 input
lowers and matches eager, and that the max pool backward reads int64
indices.
Each case fails without its part of the change.
full, full_like, arange and scalar_tensor have values that are known
ahead of time, so their QNN builders make static tensors that the
consumers read directly. When such a value is also an output of the
delegated graph, for example because the op that uses it stays on CPU,
QNN never publishes it. QNN only exposes a graph output that an op
writes, and a static tensor has no writer. The program then fails when
it runs:
y = x * 2
return y, torch.full_like(y, -1)
Method forward: the QNN context binary binds 2 tensors (1 bindable
inputs, 1 bindable outputs) but ExecuTorch passed 3 arguments
When the constant is also used inside the delegate, QNN fails to build
the graph instead ("Failed to generate Qnn context binary").
The 4 builders now share a NodeVisitor.define_constant helper. When the
value stays inside the graph, it defines the static tensor as before.
When the value is a graph output, it also adds a Reshape that copies the
static tensor into the output tensor, made with
define_custom_tensor_wrapper, which can now also build a tensor that
holds given data.
Graphs where these values stay inside the delegate are built as before.
… full
A full_like reads only the shape of its input, and the QNN backend
builds it as a static tensor. When the full_like is the only output of a
delegate, QNN drops the ops that only fed it, and the QNN graph is left
with no input, while ExecuTorch still passes one, so the program fails
when it runs:
return torch.full_like(torch.frac(x), 0.5)
Execute with 1 in, 1 out; graph expects 0 in and 1 out
With another output that reads an input, like y * 2, the program runs,
but the delegate still takes x and the ops that only fed the full_like.
A new to_edge pass, ConvertFullLikeToFull, replaces a full_like whose
shape is static with a full of the same shape, value and dtype, and
removes the ops that only fed it. On HTP, a full alone reads nothing at
runtime, so it stays on CPU, as other constant-only partitions already
do. This also covers a full_like of a value computed on CPU. A
full_like with a dynamic shape is left as it is.
Tests: a unit test checks that the full_like becomes a full and nothing
reads x any more, and that a dynamic shape keeps the full_like. It also
runs the result against eager, with a float and an int fill value. A
delegate test lowers the 2 cases above and checks their outputs on the
x86 simulator, and a quantized one lowers the full_like alone. The
cases with only the full_like fail without the pass. Other backends do
not drop a partition that reads nothing at runtime, so there the lone
full keeps its delegate.
The QNN SDK's HTP op validator sometimes rejected a delegated integer
op, like an int32 mul, with an error whose bound changed from run to
run:
[4294967295] has incorrect Value 1, expected < 225764768
Every tensor without quantization, such as an int32 or float16 one,
gets its quantize params from UndefinedQuantizeParamsWrapper. Its
CreateQuantizeParams() declared Qnn_QuantizeParams_t without an
initializer and set only the encoding definition and type, so the
encoding union kept leftover stack bytes. The validator reads them even
when the encoding is undefined, and the bound in the error is that
garbage.
All 6 CreateQuantizeParams() now value-initialize the struct, which
zeroes all of it, union included. QNN_QUANTIZE_PARAMS_INIT is not used,
because brace initialization sets only the first union member, and
newer compilers, like GCC 15, no longer zero the rest. The other 5 set
their union member right after, so they are unchanged in practice.
Test: with the QNN 2.51 x86 simulator for SM8650 in float16, one fresh
process per run, lowering a k * k of int32 k was rejected in 18 of 112
runs, and torch.where(k > 0, k, k * 2) in 8 of 48. With the union zeroed
where the params reach QNN, both passed every run (96
and 48). In each of the 11 failing processes checked, the same op
passed validation once only its quantize params union was zeroed. No unit test is added: the
failure depends on what the stack holds, so a test cannot fail
reliably without the fix.
During decomposition, exir calls a partitioner's ops_to_not_decompose filter on the nodes of the ATen program one at a time, in graph order. Each node whose op is in the list and that the filter accepts gets its target replaced right away by a copy in the EDGE_DO_NOT_DECOMP namespace (except ops that exir does not preserve, such as mutable and aliasing ops), and the original target comes back only after that partitioner's decomposition. So a filter that looks at other nodes, for example to ask whether an input comes from a dequantize op, sees that copy for every earlier node that was replaced, not the original op. The docstring said which program each filter is applied to, but not this. Partitioner authors now find it there instead of in exir's code.
A constant op such as full can end up as an output of a QNN partition,
either because it is a graph output or because the op that reads it
stays on CPU. QNN then writes it with a Reshape from a static tensor
(see define_constant). Constant ops skip the QNN op check, and HTP
cannot build such a constant with more than 5 dims, so the whole graph
fails to build:
return x * 2, torch.zeros(1, 2, 1, 1, 3, 4)
OpDef_Const failed to generate tensor
Failed to add node to Qnn Graph with error: 6007
AssertionError: Failed to generate Qnn context binary.
Before constant outputs were written with a Reshape, the same program
built, but it failed when it ran, because QNN never published the
constant.
On HTP, the partitioner now leaves constant ops of more than 5 dims on
CPU. A new test lowers a 6-dim constant that is a graph output, and one
that a CPU op reads, and checks that the multiply is delegated and the
constant is not. Both cases fail without this change.
In HTP float16, Concat can read a static bool input of all True as
False, with no error. For example, with a positive x of shape (2, 3), a
mask built from a computed mask and a constant one comes back wrong on
the x86 simulator of QNN SDK 2.51:
torch.cat([x > 0, torch.full((2, 2), True)], dim=1)
# QNN: [[True, True, True, False, False], ...]
# eager: [[True, True, True, True, True], ...]
The static tensor ExecuTorch gives QNN holds the right values: QNN
Saver records 0x01 for each element. The wrong result also depends on
the shapes and on where the static input is. At the same shapes,
constants with mixed values, float and int32 cats with a constant, and
quantized graphs come out right.
In HTP float16, the partitioner now keeps on CPU a bool cat that reads
a static tensor: a constant op such as full, or a parameter, a buffer
that the model does not update, or a lifted constant. Since the failing
shapes are not known in advance, this also moves some cats that came
out right, which can split a graph into 2 delegates. A constant op such
as full can then become a partition output, which QNN writes with a
Reshape. The partitioner keeps the fp16 HTP check in one flag,
is_htp_fp16, which the dilated conv VTCM warning now reads too.
A new test checks that, in HTP float16, a bool cat of a full or of a
buffer stays on CPU, that the same float32 cat is delegated, and that a
quantized bool cat stays delegated. Both cat delegate tests gain a bool
cat of a full. The floating-point one comes out wrong without this
change, and the quantized one checks the values of the cat that stays
delegated.
…n I64toI32
I64toI32 keeps the int64 results of ops that QNN has no builder for,
since they run on CPU. But some ops that do have a builder also stay on
CPU, and their int64 results were still cast to int32, which wraps:
x.sum(-1) # int32 input [[2**30, 2**30]], sum in skip_node_op_set
# before: -2147483648, the sum wrapped around in the int32 cast
# after: 2147483648, as in eager
The same happened to an op the partitioner skips by node name, and to
an op that reads float64, such as the `.long()` in
`(x.double() * x.double()).long()`.
I64toI32 now takes the partitioner's skip_node_id_set and
skip_node_op_set, which the pass manager already hands to passes that
ask for them. Before it rewrites the graph, it marks the ops that stay
on CPU for one of these reasons, using must_stay_on_cpu for the data
types and shapes, and treats them like ops without a builder.
As in RecomposeRmsNorm, a node name in skip_node_id_set is matched
against the graph the pass receives.
…r int64 inputs down
I64toI32 keeps the int64 result of an op that stays on CPU, so a
product, for example, does not wrap. But the inputs of such an op could
still reach it as int32, and then the value wrapped with no error:
- When a CPU op read such a result together with an int32 tensor, the
pass cast the int64 result down to int32 first, so that ops like
masked_scatter get matching dtypes:
x = torch.tensor([100000, 100000, 3, 1])
torch.fmod(x.prod(0, keepdim=True), x.sum(0, keepdim=True))
# eager: [12] before this change: [-169780]
The same happened to x.prod(-1).index_select(0, index) with int32
indices when index_select is skipped: 65536 * 65536 came back as 0.
- A CPU op read an int64 graph input or buffer through the int32 cast
made for QNN ops. x.prod(-1) with x = [[2**33, 3]] returned 0.
- A CPU op read the int32 cast of a QNN result that was int64. With mul
skipped, s * s * s * s for s = (x > 0).sum(-1) wrapped at 300**4.
A CPU op now reads every value that was int64 in the original graph as
int64. It reads an int64 input itself, not its int32 cast, and it reads
a QNN result that is int32 only because of the pass cast back up, which
loses nothing. Its other inputs keep their dtype. A cast up whose input comes
from QNN joins its partition, like the index casts the pass already
adds, and a cast with nothing else around it stays on CPU.
Each cast between QNN and CPU now takes the meta of the value it casts,
not of the op that first reads it, and the pass drops its internal
QCOM_KEEP_INT64 mark once it is done.
The CPU op test gains the fmod case and an int64 input, the test of ops
kept on CPU gains the skipped index_select and the skipped muls, and a
new test checks the meta of the casts. All fail without this change.
The int64 result test also gains a masked_scatter whose source is k + k,
to check that both inputs still get the same dtype.
ConvertFullLikeToFull turned every full_like of a static shape into a
new full node. A full_like named in skip_node_id_set or skip_node_op_set
then lost its skip, since the new node has another name and op. I64toI32
runs later and keeps int64 only for skipped ops, so it cast the full to
int32 and back, and a large fill wrapped:
torch.full_like(x, 2**40, dtype=torch.int64) # x is float32
# with full_like in skip_node_op_set: 0 instead of 1099511627776
The pass now takes both skip sets, which the pass manager already hands
to every pass that accepts them, and leaves a skipped full_like alone.
It stays on CPU, so the problem the pass solves, QNN dropping the ops
that only feed a delegate output, does not apply to it.
A new test runs the pass and I64toI32 with each skip set and checks that
the full_like stays and that the fill matches eager.
…ts only in I64toI32 _record_original_output_dtype stored a list of dtypes for a graph output whose value is a tuple, such as a returned topk. Its only reader, _preserve_output_dtype, skips such nodes, because each of their getitems is a graph output too and is cast back on its own. So the list was never read. The original dtype is now recorded only for tensor outputs, and the tuple check in _preserve_output_dtype goes away with it. The test that returns a topk of an int64 sum still matches eager for both outputs.
The ops_to_not_decompose docstring said that exir calls the filter on the nodes in graph order. exir walks all top-level nodes first and only then the nodes of each cond, map and scan submodule. So a filter on a submodule node already sees every replaced top-level node, including those after the cond. The docstring now says that, and that a filter sees the EDGE_DO_NOT_DECOMP op for each node already replaced before it.
…mposeRmsNorm RecomposeRmsNorm erased each rebuilt norm and its dead inputs with its own helper, _erase_unused. The pass already removes dead code with graph.eliminate_dead_code() before it starts, so right after a rebuild the only dead nodes are the ones that rebuild left. Calling eliminate_dead_code() there does the same thing, and dead nodes are erased one way. It still runs right away, so a node that another norm shares has the right users when _is_skipped counts them. The RMSNorm tests in test_passes.py pass unchanged.
…or on CPU
In HTP float16, the partitioner keeps on CPU a bool cat that reads a
static tensor, because HTP Concat can read such an input of all True as
False. The check looked only at the cat's direct inputs. A static tensor
reached through a reshape was missed, so the cat stayed delegated, and
on the QNN x86 simulator it came back wrong in the same way:
torch.cat([x > 0, mask.reshape(1, 4)], 1) # mask is a bool buffer
# QNN: [1, 0, 1, 0, 0, 0, 0, 0]
# eager: [1, 0, 1, 0, 1, 1, 1, 1]
The check now looks through view_copy, squeeze_copy, unsqueeze_copy,
permute_copy and expand_copy, which only change the shape or the order
of the dims, to the tensor they read. A permute or an expand of a static
tensor read right at the shapes tried on the simulator, but whether
Concat misreads depends on the shapes, so those cats stay on CPU too,
like a cat of the static tensor itself.
The partitioner test gains a view and an expand of a full, and a
reshape and a permute of a buffer. All 4 stay delegated without this
change.
On HTP, the partitioner leaves a partition on CPU when it reads no runtime value. It merges partitions even when no data flows between them, unless that makes a cycle, and it counts the output of a CPU op as a runtime value. I64toI32 and RecomposeRmsNorm each guessed this in their own way, and the guesses were wrong in some graphs: - I64toI32 cast the int64 result of such an op to int32 on CPU. For example torch.full_like(x, 2**40, dtype=torch.int64) became a full that HTP left on CPU, and it returned 0 with no error. - RecomposeRmsNorm did not rebuild a norm of constants that the partitioner merges with unrelated ops that read an input, or a norm whose input is the output of a CPU op. The partitioner still delegated the decomposed norm, and in float16 it overflowed with no error. The partitioner already estimates this for the ops it keeps whole. That estimate moves to find_nodes_left_on_cpu in partition/utils.py, and both passes use it too, with may_be_delegated telling which ops the partitioner may take. I64toI32 keeps int64 for every op that the partitioner may not take, also an op outside ATen that has no builder, and on HTP for every op the estimate leaves on CPU. Ops that the op packages in the compiler specs build count as ops QNN takes. QNN's op check is not known this early, so an op it rejects is still cast. Also: - capture_program takes skip_node_id_set and skip_node_op_set and passes them to the edge passes, like to_edge_transform_and_lower_to_qnn. - The mutable buffer log says that buffers that are not floating point stay on CPU, as the code does. - The RMSNorm runtime value tests run the export pipeline first, as the real lowering does, so pow has a lifted exponent and a QNN builder. The buffer update in one of them is a clamp_, because the pipeline turns a fill_ into a full that no longer reads the buffer. A norm of constants is tested in a graph with no other runtime value, and a norm next to a mask index is now expected to be rebuilt, since the partitioner delegates it with the comparison.
… the passes
RecomposeRmsNorm and I64toI32 estimate which ops the partitioner leaves
on CPU. They asked may_be_delegated about each op, and it did not know
two rules that QnnOperatorSupport applies on HTP: a constant of more
than 5 dims stays on CPU, and in float16 so does a bool cat of a static
tensor. So the passes could expect such an op on QNN while the
partitioner kept it on CPU. For example, a norm of constants that
reaches the input only through such a cat was rebuilt as an rms_norm.
HTP then dropped its partition, which read no runtime value, and
lowering failed, since rms_norm has no CPU kernel:
Node aten_rms_norm_default ... was not decomposed or delegated
The two rules move to get_htp_skip_reason in partition/utils.py. Both
QnnOperatorSupport and may_be_delegated call it, so they apply the same
rules. get_backend_info reads the backend, the float16 precision and
the op package ops from the compiler specs, for both passes.
QNN's op check and the limits of older HTP architectures still run only
in the partitioner.
The new test lowers such a model with ConstantFolding off, since it
would fold the norm of constants away, and fails without this change.
… in ConvertFullLikeToFull
ConvertFullLikeToFull gave the new full the dtype of the full_like but
not its device, so the full was made on the default device. With an
input on another device, such as CUDA, x + torch.full_like(x, 0.5) then
failed in the next pass that ran the graph:
FakeTensorDeviceMismatchError: Expected all tensors to be on the same
device, but found at least two devices, cuda:0 and cpu!
The full now gets the device too.
A full is also always contiguous, while a full_like takes the dim order
of its input, and the pass kept the meta of the full_like. For a
channels last x, x + torch.full_like(x, 0.5) with the add on CPU then
lowered, but failed at runtime:
Check failed (tensors_have_same_dim_order(a, b, out))
A full_like that is not contiguous is now kept.
…s in the test lowering stage ToEdgeTransformAndLower in the backend tester passed its own default compiler specs to the edge passes, and not the skip sets of its partitioners. With a QnnPartitioner that skips rms_norm, RecomposeRmsNorm still built an rms_norm, the partitioner left it on CPU, and lowering failed, since rms_norm has no CPU kernel. And with a float16 QnnPartitioner on a stage made for quantized HTP, the passes did not rebuild the float32 part of a norm, so it overflowed in float16. The stage now passes the specs of its first QNN partitioner and the skip sets of all of them, as to_edge_transform_and_lower_to_qnn does.
In float16 mode QNN runs every delegated float32 op in float16. A
timestep or rotary embedding takes sin and cos of large angles, like
t*freqs or positions/timescale, and in float16 these come out badly
wrong, for 2 reasons. float16 cannot hold the angle: its values are 1.0
apart from 1024 to 2048, and a timestep embedding with a 4e-3 period
reaches 1571. And HTP's float16 Sin and Cos lose more on top of that for
large angles. On the QNN SDK's x86 simulator for SM8650, the largest
error against float32 was 1.39 for timestep embedding angles of 1000 to
2000, and 0.50 for rotary embedding angles of 300 to 520 (positions up
to 511). A float32 sin of the float16 angle is still off by 0.95 in the
first case, so a better Sin alone would not fix it.
A new to_edge pass, TagPositionalSinCos, tags such a float32 sin or cos
for CPU fallback, together with the ops that compute its angle, so they
compute in float32 as the model does. The partitioner already leaves
nodes tagged QCOM_FALLBACK_NODE on CPU. The pass acts for HTP in float16
precision only, and needs the compiler specs, which
to_edge_transform_and_lower_to_qnn, transform_for_to_edge_pipeline (used
by export_llama), the QNN recipe and the QNN tester give it. Without
them, as in the deprecated capture_program, it does nothing and logs a
debug message. The HTP pass manager always runs it after every other
pass, last before ResolveDebugHandle, so it sees the graph the
partitioner sees. Like ResolveDebugHandle, it is moved there whatever
dep_table says. It is on by default, logs a warning when it tags ops,
and can be turned off in passes_job, for example to keep the delegated
graph of an older program, whose runtime may not have the CPU kernels
these ops need:
passes_job = get_qnn_pass_manager_cls().get_capture_program_passes()
passes_job[TagPositionalSinCos][QCOM_PASS_ACTIVATE_KEY] = False
to_edge_transform_and_lower_to_qnn(
module, inputs, specs, passes_job=passes_job
)
The QNN FP16 recipe takes the same passes_job, one job for every
method. The Qualcomm backend docs describe the CPU kernels this needs
and how to turn the pass off.
It applies only when the angle is built from graph inputs, constants,
integer positions and sizes (like position ids or x.shape[1]) by cheap
elementwise and shape ops: arithmetic, pow, exp, log, casts, views,
slices, expand, permute, cat, the arange and where that a linspace
becomes, and the full and cumsum of float positions like
ones(n).cumsum(0). The sin or cos must also be larger than the graph
inputs, and the integers computed from one, that it and its float ops
read, taken together, as an angle that spreads a few positions or
timesteps over a frequency axis is. Fourier features of a graph input,
like sin(x[..., None] * freqs), qualify too. Parameters, buffers the
graph does not change and integers built only from constants and sizes
(an int input counts as a size), like the arange inside a linspace, do
not count. A buffer the graph changes, directly, through a view or as an
out= argument, counts as an input. So a single timestep still qualifies,
while sin(x * 30) of an image, float or integer, of a cat of inputs or
of a mutable state buffer, is not tagged.
A sin or cos stays delegated when it, or an op of its angle, has more
than 262144 elements, so that no large computation, like Fourier
features of a large image, moves to CPU. The limit holds for each sin or
cos on its own, not for all the tagged ops together. That is 1024
positions times a head size of 256: the largest rotary table of the
Llama, Qwen, Gemma and PaliGemma style models at the 1024 token context
of the Qualcomm LLM examples (Llama 3.2 3B and Qwen3 have 131072, Gemma
3 1B 262144, SmolVLA 7712 to 9312). On a Cortex-A78 core a float32 sin
of that size takes about 2.5 ms. A dynamic size counts with its largest
value, so one without a max is not tagged, with a debug message.
Integer positions built only from graph inputs and constants, like
attention_mask.cumsum(-1), mask.sum(-1) + ones(n).cumsum(0) or arange(n)
< k, move to CPU with the angle, including their casts and comparisons.
This matters because I64toI32 puts an int32 cast after each int64 op,
and QNN gives that op the int32 output: a cast left on CPU behind a
delegated int64 op would read the wrong data. Integers computed from
activations stay delegated, and so does any angle built from them, so
every tagged op reads only graph inputs and constants and no delegate is
split. The walk over integers uses an explicit stack, so a long integer
chain does not reach Python's recursion limit.
A sin or cos of an activation, for example after a linear or a conv,
stays delegated, so no heavy op moves to CPU. So does a sin or cos of a
graph input that the graph only moves or casts: the graph does not
compute that angle, so the caller can keep it small. Quantized nodes,
the quantized mode and other backends are not affected.
A rotary embedding that builds its angle with a matmul, as older
Hugging Face Transformers versions do with inv_freq @ position_ids, is
not covered. Transformers 5.22 builds it with a broadcast multiply,
which is. QNN keeps matmul whole for HTP, and a whole matmul has no CPU
kernel, so it cannot simply stay on CPU. Integer positions spread over
the frequency axis before their cast to float, like
(pos[..., None] * arange(n)).float(), are not covered either: the cast
reads as many elements as the angle has, so it does not count as
positional.
The new pass reads the backend from the compiler specs with
get_backend_info and counts runtime inputs with get_runtime_inputs, as
RecomposeRmsNorm does.
Results on the simulator:
- For the 2 cases above the largest error is now 0.0002, the float16
rounding of the delegated cat after them, with the same number of
delegates.
- SmolVLA, whose language model builds rotary angles from position ids
this way: the largest error of the keys it caches drops from 0.79 to
0.19, with the same number of delegates. With a fix for its uint8
camera input cast also applied, the end to end action check drops
from 0.077 to 0.033.
Tests:
- A delegate test lowers a sinusoidal embedding with angles up to 1571,
built with linspace inside forward, for a batch of timesteps and for a
single one, and checks its output.
- Another lowers rotary angles from attention_mask.cumsum(-1) positions
plus an int64 constant made in forward, up to 500, and checks its
output, so the integer ops and their int32 casts run on CPU end to
end. A second case also returns the positions and feeds them to a
delegated op.
- A test in the backends/qualcomm/tests/rework pass tests checks that
float16 HTP tags the sin, the cos and their angle of a timestep
embedding, but not the cat after them, and that quantized HTP tags
nothing.
- A partitioner test checks that such a sin and its angle go to CPU
while the cos of a linear stays delegated, in one delegate, and that
turning the pass off delegates the sin again.
- Unit tests, which do not initialize the QNN backend, run the QNN
to_edge passes: a linspace timestep angle, an exp(-log(10000) * i)
frequency, a batch of one with buffer or linspace frequencies, an
angle from an int64 cumsum, from integer or float offset positions or
from a count over a mask as large as the angle, and aranges of a
dynamic size or int input, with or without a frequency axis, are
tagged with every op they come from, and nothing else is tagged; an
angle computed from a linear or from integers computed from an
activation (including a float sum), sin(x * 30) of a float or integer
graph input or of a cat of inputs, a sin of a graph input or of its
views, a float16 sin, quantized nodes, quantized HTP and the GPU
backend are not. Others check that a sin or cos of 262144 elements is
tagged and one of 262400 is not, also with a dynamic size bounded by
either, that a small cos sliced from a larger table and a dynamic size
without a max are not; that a mutable buffer, written in place,
through a slice or a split, or as an out= argument, counts as an
input, both in a program from torch.export and in an edge program;
that an angle behind 1000 chained integer adds is tagged; that the
pass logs a debug message without compiler specs; and that the QNN
FP16 recipe turns the pass off with the same passes_job and rejects
one that is not a dict or that is keyed by method name. Another checks
that a dep_table with no entry for the pass, like the base one, or one
that orders a pass after it or after ResolveDebugHandle, still runs it
last before ResolveDebugHandle.
to_edge_transform_and_lower_to_qnn, the QNN FP16 recipe and the other
QNN entry points run only the active passes listed in passes_job, and
the pass manager moves ResolveDebugHandle (on LPAI, followed by
LpaiPartitionFallbackSupport) to the end. A passes_job that lists only
the passes to change, like
passes_job = {SomePass: {QCOM_PASS_ACTIVATE_KEY: False}}
leaves ResolveDebugHandle out, and failed there with a bare
"list.remove(x): x not in list" that did not say how to build a valid
job. A passes_job that turns ResolveDebugHandle off failed later with
an AssertionError. An entry that only turns a pass on, like
{QCOM_PASS_ACTIVATE_KEY: True}, failed with a KeyError while building
that pass, and so did an entry with no activation key. An entry that is
not a dict failed with a TypeError, also when convert_linear_to_conv2d
turned that pass on.
Before any pass is built, the pass manager now checks that every key of
passes_job is a pass (a class or a callable like functools.partial, not
a method name), that every entry is a dict with an activation key,
that ResolveDebugHandle (on LPAI, LpaiPartitionFallbackSupport too) is
turned on, and that each active entry has its build arguments. Each
raises a ValueError naming the dict to start from for that backend, for
example:
passes_job = QnnHtpPassManager.get_capture_program_passes()
passes_job[SomePass][QCOM_PASS_ACTIVATE_KEY] = False
A passes_job that turns these passes on goes through as before,
including a reduced one that runs only the passes it lists and one with
a functools.partial key. The check reads the terminal passes from
_get_terminal_passes, so LPAI no longer needs its own check, and a
docstring that named the removed _validate_edge_passes now says what
the pass manager does.
Test: a passes_job without ResolveDebugHandle, an empty one, one with it
turned off, one with an entry that only turns a pass on, one with an
entry without the activation key, one with an entry that is a bool, or
one keyed by method name, raises the ValueError naming the pass manager,
also with convert_linear_to_conv2d and through the QNN recipe, and a
reduced one with ResolveDebugHandle still yields exactly its passes,
also with a functools.partial key. On LPAI, the same reduced job raises
the ValueError.
The pass manager test in backends/qualcomm/tests/rework builds the
to_edge passes with exported_program=None. Since ConstantFolding joined
the default passes, building it reads the program, so the test fails on
both HTP and LPAI with:
AttributeError: 'NoneType' object has no attribute 'run_decompositions'
The test now gives it a small exported program. No CI job runs the rework
tests, which is why this was not caught.
Test: test_pass_manager[htp] and test_pass_manager[lpai] in
tests/rework/passes/test.py fail before this change and pass after it.
580d9bc to
70abb65
Compare
|
@claude review effort=max |
|
Claude finished @shoumikhin's task in 37m 22s —— View job Review of the last 3 commits of #24601
No blockers in the last 3 commits; of the 5 minor findings, the main ones are that the 262144-element limit can't be changed through Detailed reviewPR Review: #24601SummaryThe last 3 commits:
There are no blockers; five minor issues and two wording notes are below. All 15 changed files were reviewed by reading the code. Python can't run in this job, so graph behavior comes from hand traces and no test was run. API Design
Testing
Code Quality
Specific Comments
RecommendationApprove. Nothing blocks merging. The two worth doing before users run into them are the limit argument with its warning, and the check at recipe creation. |
Builds on #24428. Only the last 3 commits are new.
Problem
In float16 mode, HTP runs every delegated float32 op in float16. Timestep and rotary embeddings take sin and cos of large angles that float16 cannot hold (from 1024 to 2048 its values are 1.0 apart). On the QNN x86 simulator, the largest error was 1.39 (timestep) and 0.50 (rotary). Now it is 0.0002 for both, with the same delegate count, and the SmolVLA cached key error drops from 0.79 to 0.19.
Fix
A new to_edge pass,
TagPositionalSinCos, tags a float32 sin or cos with a positional angle, and the ops computing that angle, withQCOM_FALLBACK_NODE, so they stay on CPU. Positional means:attention_mask.cumsum(-1)) by cheap elementwise and shape ops;sin(x * 30)does not;It acts only for HTP float16, logs a warning, and runs after every other active pass, last before
ResolveDebugHandle, whatever thedep_table. To turn it off (the QNN FP16 recipe takes the samepasses_job):The second commit checks
passes_jobbefore building passes. A job that leaves out or turns offResolveDebugHandle, or has a broken entry or key, now raises aValueErrorthat says how to fix it. The third gives the rework pass manager test a program, sinceConstantFoldingneeds one.Not covered: angles from a matmul or a float
stack, and integer positions spread over the frequency axis before their float cast.Release notes
Qualcomm HTP float16: the sin and cos of timestep and rotary embeddings, and the ops that build their angles, now run on CPU. A model lowered again needs CPU kernels like
aten::sin.out, which a runtime with a fixed operator list may lack. Build the list from the lowered model, or turn the pass off.Tests
dep_table, invalidpasses_job.tests/reworkpass test, run locally, since no CI job runstests/rework.