Skip to content

Qualcomm AI Engine Direct - Keep positional sin and cos in float32 - #24601

Open
shoumikhin wants to merge 28 commits into
pytorch:mainfrom
shoumikhin:qnn-fp16-positional-sincos
Open

shoumikhin wants to merge 28 commits into
pytorch:mainfrom
shoumikhin:qnn-fp16-positional-sincos

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Builds on #24428. Only the last 3 commits are new.

Problem

In float16 mode, HTP runs every delegated float32 op in float16. Timestep and rotary embeddings take sin and cos of large angles that float16 cannot hold (from 1024 to 2048 its values are 1.0 apart). On the QNN x86 simulator, the largest error was 1.39 (timestep) and 0.50 (rotary). Now it is 0.0002 for both, with the same delegate count, and the SmolVLA cached key error drops from 0.79 to 0.19.

Fix

A new to_edge pass, TagPositionalSinCos, tags a float32 sin or cos with a positional angle, and the ops computing that angle, with QCOM_FALLBACK_NODE, so they stay on CPU. Positional means:

  • built from inputs, constants and integer positions (like attention_mask.cumsum(-1)) by cheap elementwise and shape ops;
  • larger than what it reads from graph inputs and mutable buffers, so a timestep qualifies and sin(x * 30) does not;
  • at most 262144 elements (1024 positions times head size 256), for it and each angle op.

It acts only for HTP float16, logs a warning, and runs after every other active pass, last before ResolveDebugHandle, whatever the dep_table. To turn it off (the QNN FP16 recipe takes the same passes_job):

passes_job = get_qnn_pass_manager_cls().get_capture_program_passes()
passes_job[TagPositionalSinCos][QCOM_PASS_ACTIVATE_KEY] = False
to_edge_transform_and_lower_to_qnn(module, inputs, specs, passes_job=passes_job)

The second commit checks passes_job before building passes. A job that leaves out or turns off ResolveDebugHandle, or has a broken entry or key, now raises a ValueError that says how to fix it. The third gives the rework pass manager test a program, since ConstantFolding needs one.

Not covered: angles from a matmul or a float stack, and integer positions spread over the frequency axis before their float cast.

Release notes

Qualcomm HTP float16: the sin and cos of timestep and rotary embeddings, and the ops that build their angles, now run on CPU. A model lowered again needs CPU kernels like aten::sin.out, which a runtime with a fixed operator list may lack. Build the list from the lowered model, or turn the pass off.

Tests

  • Delegate tests: sinusoidal and rotary embeddings match PyTorch.
  • Unit tests: tagging, size limit, mutable buffers, pass order with a custom dep_table, invalid passes_job.
  • A tests/rework pass test, run locally, since no CI job runs tests/rework.

@shoumikhin
shoumikhin requested a review from psiddh as a code owner November 4, 2026 08:57
@shoumikhin shoumikhin added the release notes: qualcomm Changes to the Qualcomm backend delegate label Nov 4, 2026
Copilot AI balanced review requested due to automatic review settings November 4, 2026 08:57
@pytorch-bot

pytorch-bot Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24601

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 2 Pending

As of commit 70abb65 with merge base b69ece4 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 4, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@shoumikhin

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Claude finished @shoumikhin's task in 31m 8s —— View job


Request Changes: TagPositionalSinCos can move integer ops that I64toI32 rewrote for QNN onto CPU, which may silently corrupt the cumsum position-id case it advertises, and QNN recipe users cannot turn the default-on pass off.

Detailed review

PR Review: #24601

Summary

The new commit adds TagPositionalSinCos, a to_edge pass that is on by default. In HTP fp16 mode it sends float32 sin/cos, and the ops that compute their angle, to CPU. It also moves the HTP-fp16 check into one helper, is_htp_fp16. The main problem is a suspected interaction with I64toI32 on the integer side of the angle that could give wrong results without any error, and no test lowers that case through the real pipeline. Two smaller problems:

  • QNN recipe users cannot turn the pass off.
  • The pass silently does nothing on entry points that don't pass compiler specs.

Scope: only 5ea2021, as the description asks. I read the #24428 commits only where this commit touches them. All findings come from reading the code; I did not run anything.

Code Quality

  • _passes/tag_positional_sin_cos.py:110-119: integer nodes that I64toI32 rewrote for QNN can end up on CPU (suspected; major if it reproduces). I64toI32 runs first and rewrites the integer side of the graph on the assumption that QNN will run it. This pass then moves part of that side to CPU, in two ways:

    • Casts after int64 producers. I64toI32 adds a _to_copy(int32) after every int64 producer (i64_to_i32.py:63-69). The cast is in _ANGLE_OPS, so it gets tagged. The producer counts as an integer leaf, so it stays delegated. For cumsum, the builder gives the int64 edge tensor an int32 QNN output (builders/op_cum_sum.py:54-55). The runtime binds the int64 ExecuTorch buffer to that output as is (runtime/QnnExecuTorchBackend.cpp:203), so the CPU cast would read pairs of int32 values as one int64.
      • Trigger: the PR's own position-id case, cos(mask.cumsum(-1).unsqueeze(-1).float() * freqs), with an int64 mask, lowered through to_edge_transform_and_lower_to_qnn. This only happens if HTP accepts an int32 CumulativeSum.
      • The SmolVLA results don't rule this out: its cumsum runs on a bool mask and may never have been delegated.
    • Retyped int64 constants. I64toI32 changes int64 constants to float32 in the meta only, not in the data (i64_to_i32.py:132-135, applied at :141-145). Nothing restores the meta after partitioning, and the emitter pairs the float32 spec with the real int64 storage.
      • Trigger: sin((self.pos[:T] + start).float() * self.inv_freq) with an int64 pos buffer. This chain used to be delegated. Its tagged slice/add now runs on CPU and reads that buffer.

    Either fix avoids both cases:

    • keep the nodes I64toI32 rewrote on the QNN side, or
    • when the integer chain reads only inputs, move all of it to CPU, producer included.

    A delegate test for each trigger would show whether it reproduces. The current tests cannot catch either case (see Testing). Fix this →

  • _passes/backends/htp/qnn_htp_pass_manager.py:40-42: nothing enforces the comment "Last before ResolveDebugHandle". The only constraint is [TagQuantIO].

    • I64toI32, whose casts this pass tags, is not an ancestor of TagQuantIO (qnn_pass_manager.py:331-334, 347). Neither are the other passes that insert nodes. Today the order comes only from how solve_constraints breaks ties (first in, first out).
    • to_edge_transform_and_lower_to_qnn accepts a caller-supplied dep_table, and that table can leave the pass unconstrained (the base QnnPassManager table does). The pass then runs early. Nodes that later passes insert on the angle path stay untagged and get delegated in between CPU nodes.

    Order the pass after every other active pass, or check its position in _validate_edge_passes next to the ResolveDebugHandle check. Fix this →

  • _passes/tag_positional_sin_cos.py:92-93, 108: an angle that is only a view of a graph input gets tagged. The stated reason sin(t) stays delegated is that the graph does not compute that angle. But the code only checks whether the angle set is empty. So sin(t[:, None]) or sin(t.view(-1)) moves to CPU, even though the graph only added a view. Requiring at least one arithmetic op in the set would match the stated rule. Fix this →

Testing

  • tests/test_passes.py: the tests don't check what the pass promises.

    • Wrong input graph. test_tag_positional_sin_cos runs the pass on plain to_edge(torch.export.export(...)) (:1855), not on the graph the QNN pipeline gives it. That graph doesn't use QNN's decomposition table, which is why the test asserts on addmm (:1876) where the pipeline keeps linear. It also lacks I64toI32's casts and retyped constants. So the cumsum case is never checked against the graph the pass really sees.
    • Shallow assertions. The test only checks each sin/cos and the node right below it (:1860). A version that tags the top mul but leaves the deeper unsqueeze/cast of the positions chain untagged would still pass.
    • Untested dtype check. No test covers the float32 check (tag_positional_sin_cos.py:90); removing it breaks nothing.
    • Partitioner test. It checks only where sin ends up, not the angle ops. It doesn't count executorch_call_delegate nodes, so no test backs the "same number of delegates" claim.

    Fix this →

API Design

  • The pass only runs where compiler specs reach the to_edge pass list (needs discussion). On these entry points it silently does nothing, so the same fp16 HTP model gets different numerics depending on the API used:

    • capture_program (utils/utils.py:585-589).
    • QnnPassManager.transform_for_to_edge_pipeline (qnn_pass_manager.py:481-492). It has no compiler_specs parameter, and examples/models/llama/export_llama_lib.py:1621 uses it.
    • The QNN Tester (tests/tester.py:83).

    RecomposeRmsNorm from Qualcomm AI Engine Direct - Fix float16 RMSNorm, float64 and int64 data, and constant outputs #24428 has the same gap. Several things point to doing this in the partitioner instead:

    • QnnOperatorSupport already has the specs and calls is_htp_fp16 (qnn_partitioner.py:110).
    • The TODO at qnn_pass_manager.py:384 already asks to move to_edge passes into the partitioner.

    Choosing these nodes in QnnPartitioner, with the opt-out as a partitioner argument, would cover every entry point. It would also stop depending on pass order and on node meta set by a to_edge pass.

Backward Compatibility

  • recipes/qnn_recipe_provider.py:195: recipe users can't reach the documented opt-out. The FP16 recipe calls get_to_edge_transform_passes(ep, compiler_specs=...) with the default passes_job, and it accepts only soc_model and the skip_* kwargs (:142-145). Consequences:

    • Re-exporting an existing fp16 model through the recipe always moves its positional sin/cos chains to CPU, however large the tensors.
    • The new program needs portable kernels for sin, cos, mul, pow, arange, where, cat and others. A selective-build runtime made from the old program's op list fails at load with a missing operator.
    • Nothing above INFO level points the user to this pass. The only trace is the partitioner's "Forced Fallback" INFO line (qnn_partitioner.py:168-169).

    Expose the switch as a recipe kwarg, and log a warning that names the pass and the opt-out when it tags nodes. Fix this →

Performance

  • _passes/tag_positional_sin_cos.py:114-119: untagged integer and bool leaves can split a delegate in two (suspected). The pass never tags a leaf, even one that sits between tagged nodes and delegated ones:

    • Suppose HTP delegates attention_mask.cumsum(-1). Its output feeds the CPU chain, whose sin/cos go back into the delegate (rope applied to q/k). The partitioner then has to cut what was one delegate into two (HTP → CPU → HTP).
    • A comparison inside the angle, such as torch.where(torch.arange(d) < k, ...), can become a one-op delegate between the CPU arange and the CPU where.

    Also tagging cheap integer and bool producers whose inputs are graph inputs, constants or tagged nodes would avoid this. In the cumsum case it would also fix the first Code Quality issue.

Recommendation

Request Changes

Before merging, add a test that lowers an int64-mask cumsum angle and an int64-buffer angle through to_edge_transform_and_lower_to_qnn, and use it to rule out or fix the I64toI32 interaction. Silently wrong positions would be worse than the fp16 error this commit fixes. Recipe users also need a way to turn the pass off. The other findings are minor or open for discussion.

@shoumikhin
shoumikhin force-pushed the qnn-fp16-positional-sincos branch from 5ea2021 to 9179a13 Compare November 4, 2026 17:15
Copilot AI balanced review requested due to automatic review settings November 4, 2026 17:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@shoumikhin

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Claude finished @shoumikhin's task in 39m 39s —— View job


Request Changes: the default-on TagPositionalSinCos moves any float32 sin/cos computed from graph inputs to CPU with no size bound, which can break selective-build runtimes, and the new recipe kwarg passes_job crashes when given a partial dict; a low-confidence int64-constant issue needs an end-to-end test to settle.

Full review

PR Review: #24601

Summary

The PR adds TagPositionalSinCos, an HTP fp16 to_edge pass that is on by default. It tags float32 sin/cos whose angle is computed from graph inputs, constants and integer positions, so the partitioner keeps them on CPU in float32. It also adds a passes_job kwarg to the QNN FP16 recipe so users can turn the pass off. Three problems:

  • The tagging has no size bound and is on by default.
  • The recipe kwarg crashes on a partial dict.
  • One int64-constant path may produce wrong values (low confidence).

Scope: only the last commit, 9179a13; the other four commits belong to #24428. I reviewed by reading the code only, because this job could not run Python.

Code Quality

  • (Low confidence) A tagged CPU op may read an int64 constant as float32, because I64toI32 only retypes its meta.
    • _get_position_nodes skips placeholder args (tag_positional_sin_cos.py:185), so an angle that reads an int64 constant is still tagged.
    • For a constant that passes is_constant (builders/utils.py:176-177, which requires meta["val"].constant), I64toI32 inserts no cast. It only rewrites the placeholder's meta to float32 and leaves the data int64 (i64_to_i32.py:141-145, :135).
    • That was safe while the consumer was delegated, because get_parameter converts the data to the meta dtype (builders/utils.py:71).
    • With the consumer on CPU, the placeholder stays in the top-level program. The emitter then pairs the float32 spec with the int64 storage (exir/emit/_emitter.py:2154, :470-498).
    • I could not confirm that .constant survives run_decompositions. The likely trigger is an int64 scalar such as torch.tensor(k) created in forward and added to the positions.
    • No test runs an integer-position angle end to end. test_qnn_backend_sin_cos_large_angle (test_qnn_delegate.py:2515) only feeds a float t, and the new test_passes.py tests only inspect the graph.
    • A lower_module_and_test_output case with mask.cumsum(-1) positions plus an int64 constant offset would settle this. It would also cover the I64toI32 casts that now run on CPU. Fix this →

Backward Compatibility

  • The default-on pass tags any float32 sin/cos of an elementwise expression of graph inputs, with no size bound.
    • _get_angle_nodes skips placeholders (tag_positional_sin_cos.py:153), and call tags the sin/cos as soon as one non-view op is part of the angle (:131). Output 0 of test_tag_positional_sin_cos (sin(t * linspace(...))) depends on this.
    • Example: for conv(torch.sin(x * 30)) on a [1, 3, 512, 512] image, mul and sin now run over 786K elements in portable float32 kernels instead of on HTP. That is also a performance regression.
    • The pass is on by default (qnn_htp_pass_manager.py:30) for to_edge_transform_and_lower_to_qnn and the FP16 recipe. Re-exporting an unchanged fp16 model can therefore produce a .pte that needs CPU kernels (sin, cos, mul, cumsum, _to_copy, …). A selective-build runtime that does not link those kernels fails to load it.
    • Consider bounding the tagging, for example to sin/cos outputs under a small numel limit, which positional and timestep embeddings fit within.
    • Static angles that read no graph input could instead be folded in float32 ahead of time, which needs no CPU kernels. Fix this →

API Design

  • The recipe kwarg passes_job fails on the obvious partial override.
    • Validation only checks isinstance(passes_job, dict) (qnn_recipe_provider.py:139-144), but get_to_edge_transform_passes runs only the passes present in the dict (qnn_pass_manager.py:410-418).
    • passes_job={TagPositionalSinCos: {QCOM_PASS_ACTIVATE_KEY: False}} passes validation, then fails inside lowering with an IndexError from _validate_edge_passes (self.passes[-1] on an empty list, qnn_pass_manager.py:436-439).
    • If the dict also contains ResolveDebugHandle, every other edge pass is skipped without any error.
    • {TagPositionalSinCos: False} raises a TypeError.
    • Only a full copy of get_capture_program_passes() works, which is what the test uses (test_passes.py:2009-2010).
    • The kwarg exists to turn one pass off, so a boolean recipe kwarg would be simpler. Alternatively, merge the user dict over get_capture_program_passes() and validate its entries. Fix this →

Recommendation

Request Changes: bound the default-on tagging (or make it opt-in), and make the recipe opt-out hard to misuse. The int64-constant concern is low confidence; the suggested end-to-end test would confirm or clear it.

@shoumikhin

Copy link
Copy Markdown
Contributor Author

@qti-horodnic could you take a look? It keeps positional sin and cos in float32 on HTP, so large angles stay accurate.

@shoumikhin
shoumikhin force-pushed the qnn-fp16-positional-sincos branch from 9179a13 to 70a8bc6 Compare November 4, 2026 20:09
Copilot AI balanced review requested due to automatic review settings November 4, 2026 20:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@shoumikhin
shoumikhin force-pushed the qnn-fp16-positional-sincos branch from 70a8bc6 to 3eae0b0 Compare November 4, 2026 20:47
Copilot AI balanced review requested due to automatic review settings November 4, 2026 20:47
@shoumikhin

Copy link
Copy Markdown
Contributor Author

Thanks. Updated:

  1. Size bound. A sin or cos now moves to CPU only if its output is larger than every graph input and integer tensor its float angle ops read. Float parameters and buffers do not count. A positional angle spreads a few positions or timesteps over a frequency axis, so it passes. sin(x * 30) of an image does not, so it stays on HTP, whether the image is float or int64. The pass stays on by default, because these embeddings are wrong in float16 otherwise. It still logs a warning when it tags ops and can be turned off with passes_job. A selective build made from the program (EXECUTORCH_SELECT_OPS_MODEL) picks up the CPU kernels it needs.

  2. Recipe passes_job. It takes the same full dict as to_edge_transform_and_lower_to_qnn: passes_job = get_qnn_pass_manager_cls().get_capture_program_passes(), then passes_job[TagPositionalSinCos][QCOM_PASS_ACTIVATE_KEY] = False. A dict that leaves out passes, or whose entries are not dicts with those 2 keys, now raises a ValueError that names this form, instead of failing inside lowering. That covers the 3 cases in the review. The warning names the same form.

  3. int64 constant. I could not reproduce it. A new delegate test lowers rotary angles from mask.cumsum(-1) plus an int64 torch.tensor(200) made in forward. It matches PyTorch on the x86 simulator, and fails when the pass does not tag the angle.

@shoumikhin

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@claude

claude Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Claude finished @shoumikhin's task in 42m 26s —— View job


Request Changes: the new size bound in TagPositionalSinCos counts integers built only from constants, so a timestep embedding with a batch of one, whose frequencies come from linspace or an integer arange in forward, still computes sin and cos in float16. That includes the PR's own SinusoidalEmbedding at batch 1.

Detailed review

PR Review: #24601

Summary

The PR adds TagPositionalSinCos, an HTP float16 to_edge pass. It keeps large positional sin/cos and the ops that compute their angle on CPU in float32, with a passes_job opt-out that the QNN recipe validates.

The size bound added in this round compares the sin/cos against the wrong tensors:

  • it misses angles with a batch of one when their frequencies are computed in forward;
  • it tags all-integer angles of any size.

Only the last commit (3eae0b0) was reviewed, because the 4 before it belong to #24428. This job can run only git, so I traced the graphs below by reading the pass and the torch._refs.linspace decomposition. Nothing was run.

Code Quality

  • The size bound counts integers that carry no input data, and never checks the sin/cos's own input (backends/qualcomm/_passes/tag_positional_sin_cos.py:158-166).

    • Integers built only from constants and sizes still count. Every integer that a float angle op reads is compared. That includes the int arange and the lt mask of a decomposed linspace, and an int arange(n) frequency axis.
      • With t of shape [1, 1], the PR's SinusoidalEmbedding (models.py:2644) has 64 angles. The cast and where of its linspace read an int arange and a bool lt of 64 elements. The check 64 < 64 fails, so its sin and cos stay delegated in float16.
      • Outputs 0 and 3 of test_tag_positional_sin_cos would be rejected the same way with a batch-1 t.
      • This contradicts the commit message: "Float parameters and buffers do not count, so a batch of one timestep still qualifies".
      • The outcome also depends on dtype. At batch 1, the sin of t * torch.arange(8) is rejected, while the sin of t * torch.arange(8, dtype=torch.float32) is tagged.
      • No test covers this. The only batch-1 case is t[:1] * self.freqs, whose frequencies are a buffer, and test_qnn_backend_sin_cos_large_angle uses 4 timesteps.
    • The sin/cos's own input is never compared.
      • The sin/cos is itself a float op that reads the angle, but it is not part of the scan. An angle that stays integer all the way to the sin therefore passes at any size.
      • For example, torch.sin(x * 30) with an int64 x is tagged, together with its integer ops. This contradicts "sin(x * 30) of an image, float or integer, stays delegated".
      • The tests cover only mask.float() * 30.
    • Fix:
      • Count only user inputs and integers computed from a user input. The memoized positions sets already tell these apart: such an integer's set has a node that reads a placeholder in self.user_inputs.
      • Scan angle_nodes | {node}.
      • Add three tests:
        • torch.sin(t[:1] * torch.linspace(1, 1000, 8)), which should be tagged;
        • torch.sin(mask * 30), which should not be tagged;
        • the delegate test run with a single timestep.

    Fix this →

  • The passes_job ValueError puts the whole dict in its message (backends/qualcomm/recipes/qnn_recipe_provider.py:159-163).

    • The message lists every default pass class with its kwargs, and still does not say what is wrong.
    • The checks next to it print only type(...).
    • Name the missing passes (defaults.keys() - passes_job.keys()), or the malformed entry, instead.

    Fix this →

Performance

  • Constant angles are recomputed on CPU on every call (tag_positional_sin_cos.py:133-137). This one is optional.

    • An angle built only from parameters, buffers and constants is itself a constant, for example torch.arange(n)[:, None] * self.inv_freq with a static n.
    • Its tagged arange, mul, casts and sin/cos still run on CPU on every call, and they need CPU kernels at runtime.
    • Folding the sin/cos into a float32 constant at export gives the same precision and leaves nothing on CPU, since values in [-1, 1] survive float16 inside the delegate.
    • constant_prop_pass, which ConvertMhaToSha already calls (convert_mha_to_sha.py:622), could do the folding.

    Fix this →

Recommendation

Request Changes

The size bound exists to tell positional angles apart from data angles. As written, it leaves the PR's own SinusoidalEmbedding with its float16 error at batch 1, which is the usual inference case, and it tags integer data angles. The fix stays inside _broadcasts_inputs, plus three tests. The other two items are minor.

@shoumikhin
shoumikhin force-pushed the qnn-fp16-positional-sincos branch from 3eae0b0 to bd435fd Compare November 5, 2026 01:18
Copilot AI balanced review requested due to automatic review settings November 5, 2026 01:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@shoumikhin

Copy link
Copy Markdown
Contributor Author

Thanks. Changes in the new push:

  1. Size bound. You were right on both points. Now only graph inputs, and integers computed from them, count. Integers built only from constants and sizes, like the arange and the comparison inside a linspace, or an integer arange(n), no longer count. The input of the sin or cos itself counts too, so torch.sin(mask * 30) with an integer mask is no longer tagged. New cases in test_tag_positional_sin_cos: torch.sin(t[:1] * torch.linspace(1, 1000, 8)) is tagged, and torch.sin(mask * 30) is not. test_qnn_backend_sin_cos_large_angle now also runs SinusoidalEmbedding with a single timestep. It passes on the simulator, and fails with the old size bound.

  2. passes_job error. The recipe no longer has its own rule for passes_job. It only checks that it is a dict, and hands it to the same pass manager code as to_edge_transform_and_lower_to_qnn. When ResolveDebugHandle is missing or turned off, that code now raises a short ValueError that names the dict to start from. A test checks this on the pass manager and through the recipe.

  3. Folding constant angles. I left this out of this PR. It only helps when the angle reads no input at all. constant_prop_pass could do it, but it folds every constant expression in the program, not only these angles. That is a bigger change than this fix. The cost today is small: a float32 sin and cos table of 1024 by 32 takes about 0.3 ms per call with ExecuTorch's CPU kernels on one laptop CPU thread. If a runtime lacks those CPU kernels, the pass can be turned off at export.

@shoumikhin

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Claude finished @shoumikhin's task in 47m 19s —— View job


PR Review: #24601

Approve: no correctness problems found in b6033c8 and bd435fd; four minor items remain (where the new passes_job check lives, the recipe's passes_job not taking the per-method form, two test cases that pass for a reason other than the one they state, and no user docs for the new fp16 CPU fallback).

Detailed review

Summary

The two commits add TagPositionalSinCos, an HTP fp16 to_edge pass that keeps float32 positional sin/cos and their angle ops on CPU (with a passes_job opt-out that the QNN FP16 recipe now accepts), and make get_to_edge_transform_passes raise a ValueError when passes_job has no active ResolveDebugHandle. No correctness problems found; the four items below are minor. Only these two commits were reviewed; the other seven belong to #24428. The job allows only git commands, so nothing was run, and the findings come from reading the code.

Code Quality

  • backends/qualcomm/_passes/qnn_pass_manager.py:400-412: the new ValueError restates the base rule of _validate_edge_passes (:449-452) outside it, and it needed # noqa: C901 on get_to_edge_transform_passes (:386). Backends that override _validate_edge_passes still fail with the bare errors the commit message sets out to replace. On LPAI (backends/qualcomm/_passes/backends/lpai/qnn_lpai_pass_manager.py:59-65), a job holding only the default ResolveDebugHandle entry passes the check and then raises IndexError at self.passes[-2]. The reduced job from test_to_edge_passes_need_resolve_debug_handle ends in a bare AssertionError, since it lacks LpaiPartitionFallbackSupport. The new HTP order is not checked either. Only the dep table (backends/qualcomm/_passes/backends/htp/qnn_htp_pass_manager.py:40-45) keeps TagPositionalSinCos after I64toI32. A hand-built dep_table without those entries runs the pass first, with no error, and then the casts I64toI32 puts after int64 user inputs are left untagged. In-repo callers all start from the HTP table. Suggest raising the ValueError (with the get_capture_program_passes() hint and a length guard) from _validate_edge_passes and the LPAI override, adding a QnnHtpPassManager override that requires TagPositionalSinCos, when it ran, at passes[-2], and dropping the pre-check and the noqa.
    Fix this →

Testing

  • backends/qualcomm/tests/test_passes.py:2060: the gpu case never runs the pass. The GPU pass manager's defaults don't include TagPositionalSinCos, and the direct call at :2075-2076 only runs for the quantized cases, so this case would pass even if the pass tagged GPU graphs.
  • backends/qualcomm/tests/test_passes.py:2020: the argmax case sits under "Integers computed from an activation stay delegated" (:2014-2015), but x is a user input. It stays untagged only because argmax is not in _POSITION_OPS (backends/qualcomm/_passes/tag_positional_sin_cos.py:242). If argmax were added, its (4, 1) result would be smaller than the (4, 8) angle, so the case would be tagged even though nothing about activations changed. Give it its own comment or drop it; the linear case at :2016-2019 already covers activations.
    Fix this →

API Design

  • backends/qualcomm/recipes/qnn_recipe_provider.py:139-144, 205-213: the new passes_job recipe kwarg only takes a single job. to_edge_transform_and_lower_to_qnn also takes the per-method Dict[str, OrderedDict] form (backends/qualcomm/utils/utils.py:367, 470), but the edge pass lambda ignores method_. create_recipe(QNNRecipeType.FP16, soc_model=..., passes_job={"forward": job}) passes the isinstance(dict) check, then fails with the misleading "passes_job lacks an active ResolveDebugHandle". A multi-method export also cannot set a job per method. Either resolve passes_job[method_] when the keys are method names, or reject string keys in _validate_partitioner_parameters, and say which form is accepted next to # 5. passes_job in qnn_recipe_types.py:23.
    Fix this →

Backward Compatibility

  • docs/source/backends-qualcomm.md: on HTP fp16, positional sin/cos and their angle ops now leave the delegate by default, so the runtime needs their CPU kernels. The change and its opt-out (passes_job[TagPositionalSinCos][QCOM_PASS_ACTIVATE_KEY] = False, now also a recipe kwarg) are described only in the pass docstring and a log warning. A re-exported model then fails to load with a missing operator on a runtime built from a hand-written op list, and nothing in the docs explains why. Add a short entry with the opt-out snippet next to the fp16 troubleshooting items (around :690).
    Fix this →

Recommendation

Approve: nothing blocks merging. The four items are minor and can be handled here or in a follow-up.

  • Gather context (PR description, author replies, commit range)
  • Run three independent reviews of the last 2 commits (b6033c8, bd435fd)
  • Consolidate findings
  • Fact-check findings against the code
  • Post review

The backend mapped torch.float64 to QNN_DATATYPE_FLOAT_32 and
delegated float64 ops. The data was never
converted: float64 constants and graph inputs reached QNN as raw 8-byte
values read as 4-byte floats. So float64 math gave wrong results with no
error. For example, (x.double() * 2 + 1).float() returned zeros on the
QNN SDK's x86 simulator for SM8650 in float16 mode, and a sinusoidal
timestep embedding computed in float64, as openpi and LeRobot's pi0,
pi05 and SmolVLA do, was off by up to 2.0.

float64 is now left out of the QNN type map, as bfloat16 already is. A
node that reads or writes a float64 tensor stays on CPU, where float64
is computed exactly. Ops on other dtypes are delegated as before. The
quantizer's dtype check reads the same map, so it now treats a float64
input as having no QNN type instead of failing on the missing key.

The To builder's uint8 cast through int32 on HTP no longer needs a
float64 case, since a float64 cast now stays on CPU.

On the simulator, both examples above now match PyTorch exactly, and
the error of SmolVLA's denoising step drops from 0.32 to 0.0027
(largest absolute error, outputs up to 3.2).

New tests lower float64 math between float32 ops, unquantized and
quantized, and check the output.
When the QNN partitioner delegates a mutable buffer, the delegate owns
the buffer's memory. The QNN runtime does not load the buffer's starting
value into that memory yet, so the buffer starts from whatever the
memory held. For an int64 counter the result is wrong with no error:

    self.register_buffer("count", torch.zeros(2, dtype=torch.long))
    ...
    self.count.add_(1)
    # count comes back as [181093535, 1] instead of [1, 1]; the first
    # value changes from run to run

The partitioner now leaves integer mutable buffers on CPU, and any
other buffer that is not floating point, even with the default
skip_mutable_buffer=False. The ops that update the buffer can still run
in the delegate: the buffer goes in as a delegate input, and the new
value comes back as a delegate output that CPU writes to the buffer, as
skip_mutable_buffer=True already does for every buffer. Floating-point
buffers are delegated as before, so a float counter still starts from
leftover memory.

This matters more with the next change, which casts every int64 result
to int32: then an update such as count.add_((x > 0.5).sum(-1)), which
runs on CPU today for an int64 or an int32 count, could be delegated
together with its buffer.

A test lowers int64 and int32 counters updated by a constant and an
argmax, and checks that only the float buffer is delegated.
Many QNN builders read and write integer tensors, such as indices,
as int32. I64toI32 turns int64 inputs and constants into int32, and it
added an int32 cast after the ops in a fixed list: argmax, argmin, arange, cumsum, full, scalar_tensor and
casts. Other ops make int64 too, like a sum or a product of bools or
integers, and the indices of topk and sort. Their results stayed int64,
which caused lowering errors and wrong results:

- An op that needs equal dtypes failed when the pass traced the graph
  again:

      (x > 0.5).sum(-1).masked_scatter(mask, k)
      RuntimeError: masked_scatter: expected self and source to have
      same dtypes but got torch.int64 and torch.int32

- A value that went into or out of QNN this way was read with the
  wrong size, with no error. For example x[:, (x > 0.5).sum(-1)]
  picked the wrong columns, and topk indices [1, 2] came back as
  8589934593.

The pass now casts every int64 result to int32, also the ones read
through getitem, instead of relying on a list. The int64 casts that the
pass adds for the index of gather and scatter stay, because the CPU
kernels need int64 there. The 8-bit quantized embedding now gets the
same index cast, since its CPU kernel only takes int64. An input listed
in skip_node, like the token ids of the Llama flow, is still int64, so
it gets no index cast. Int64 graph outputs are still cast back to int64
at the end. For a multi-output op such as topk, whose input can now be
int32, each returned output is cast back through its getitem.

This also changes graphs that worked before, when they have such an
int64 result. A value that does not fit in int32 now wraps around, as
already happens to int64 inputs and to the listed ops. When the op that
makes the value stays on CPU, the new cast can become a QNN delegate
that only converts the dtype, which the next change keeps on CPU, or
split an existing delegate in 2.
The previous change keeps integer mutable buffers on CPU, so a buffer
updated from such a value still gets the right value. Graphs whose int64
results all come from the listed ops are unchanged.

The debug handle test model, in test_passes and in the rework pass
tests, gains one node, the cast of its topk indices, so the handles
after it move up by 1.

The test for integer mutable buffers also updates the counters by a sum
of bools now, an int64 result that this change casts.
… CPU

I64toI32 casts int64 graph inputs and int64 results to int32, because
QNN handles integer tensors as int32, and casts int64 graph outputs
back. When the ops around such a cast stay on CPU, the casts can end up
in a partition of their own. QNN then gets a delegate that only converts
dtypes, which costs a graph execution on every inference and does no
work of the model. For example:

    (x > 0.5).sum(-1).masked_scatter(mask, k)
    # sum and masked_scatter run on CPU
    # before: 3 delegates, [gt], [cast k, cast sum], [cast the output back]
    # after:  1 delegate, [gt]

I64toI32 now marks every cast it inserts, and the QNN partitioner keeps
a partition made only of such casts on CPU. A partition that holds a
marked cast next to other ops is unchanged, and so is a cast the model
makes itself.

A new test checks that this graph lowers to the gt delegate alone and
that a model that is only a cast is still delegated. Another checks that
only the mark counts, not the op, since other passes insert _to_copy
too. The delegate tests of Int64Results now expect 2 delegates instead
of 3.
…n I64toI32

I64toI32 casts every int64 result to int32, because QNN handles integer
tensors as int32. Some ops have no QNN builder at all, so they always
run on CPU, where int64 works. For them the cast only did harm:

    x.prod(-1)    # int32 input [[65536, 65536]]
    # before: 0, the product wrapped around in the int32 cast
    # after:  4294967296, as in eager

    s, zp = choose_qparams_per_token_asymmetric(x, torch.int8)
    quantize_per_token(x, s, zp, -128, 127, torch.int8)
    # before: the runtime aborted, since the CPU kernel needs an int64
    #         zero point and got the int32 cast
    # after:  runs and matches eager

The int64 results of an ATen or quantized_decomposed op that QNN has no
builder for now stay int64. A QNN op that reads such a result gets an
int32 cast of it, so int64 still never enters QNN. Ops with a builder,
and ops of other namespaces such as custom ops, are cast as before, and
a CPU op reads their results through that cast, as before. A CPU op that
reads an int32 tensor together with a kept int64 result reads the int32
cast of that result too, so that an op like masked_scatter gets
matching dtypes. A value that a QNN op reads still wraps around when it
does not fit in int32.

max_pool2d_with_indices_backward runs on CPU too, and its kernel needs
int64 indices, which the pass now casts to int32 when they come from a
getitem. It gets the same int64 index cast as gather and scatter.

A test checks that a product keeps its int64 value, that the CPU
quantize_per_token reads an int64 zero point, that an index op reads a
product as int32, that masked_scatter of a product into an int64 input
lowers and matches eager, and that the max pool backward reads int64
indices.
Each case fails without its part of the change.
full, full_like, arange and scalar_tensor have values that are known
ahead of time, so their QNN builders make static tensors that the
consumers read directly. When such a value is also an output of the
delegated graph, for example because the op that uses it stays on CPU,
QNN never publishes it. QNN only exposes a graph output that an op
writes, and a static tensor has no writer. The program then fails when
it runs:

    y = x * 2
    return y, torch.full_like(y, -1)

    Method forward: the QNN context binary binds 2 tensors (1 bindable
    inputs, 1 bindable outputs) but ExecuTorch passed 3 arguments

When the constant is also used inside the delegate, QNN fails to build
the graph instead ("Failed to generate Qnn context binary").

The 4 builders now share a NodeVisitor.define_constant helper. When the
value stays inside the graph, it defines the static tensor as before.
When the value is a graph output, it also adds a Reshape that copies the
static tensor into the output tensor, made with
define_custom_tensor_wrapper, which can now also build a tensor that
holds given data.
Graphs where these values stay inside the delegate are built as before.
… full

A full_like reads only the shape of its input, and the QNN backend
builds it as a static tensor. When the full_like is the only output of a
delegate, QNN drops the ops that only fed it, and the QNN graph is left
with no input, while ExecuTorch still passes one, so the program fails
when it runs:

    return torch.full_like(torch.frac(x), 0.5)

    Execute with 1 in, 1 out; graph expects 0 in and 1 out

With another output that reads an input, like y * 2, the program runs,
but the delegate still takes x and the ops that only fed the full_like.

A new to_edge pass, ConvertFullLikeToFull, replaces a full_like whose
shape is static with a full of the same shape, value and dtype, and
removes the ops that only fed it. On HTP, a full alone reads nothing at
runtime, so it stays on CPU, as other constant-only partitions already
do. This also covers a full_like of a value computed on CPU. A
full_like with a dynamic shape is left as it is.

Tests: a unit test checks that the full_like becomes a full and nothing
reads x any more, and that a dynamic shape keeps the full_like. It also
runs the result against eager, with a float and an int fill value. A
delegate test lowers the 2 cases above and checks their outputs on the
x86 simulator, and a quantized one lowers the full_like alone. The
cases with only the full_like fail without the pass. Other backends do
not drop a partition that reads nothing at runtime, so there the lone
full keeps its delegate.
The QNN SDK's HTP op validator sometimes rejected a delegated integer
op, like an int32 mul, with an error whose bound changed from run to
run:

    [4294967295] has incorrect Value 1, expected < 225764768

Every tensor without quantization, such as an int32 or float16 one,
gets its quantize params from UndefinedQuantizeParamsWrapper. Its
CreateQuantizeParams() declared Qnn_QuantizeParams_t without an
initializer and set only the encoding definition and type, so the
encoding union kept leftover stack bytes. The validator reads them even
when the encoding is undefined, and the bound in the error is that
garbage.

All 6 CreateQuantizeParams() now value-initialize the struct, which
zeroes all of it, union included. QNN_QUANTIZE_PARAMS_INIT is not used,
because brace initialization sets only the first union member, and
newer compilers, like GCC 15, no longer zero the rest. The other 5 set
their union member right after, so they are unchanged in practice.

Test: with the QNN 2.51 x86 simulator for SM8650 in float16, one fresh
process per run, lowering a k * k of int32 k was rejected in 18 of 112
runs, and torch.where(k > 0, k, k * 2) in 8 of 48. With the union zeroed
where the params reach QNN, both passed every run (96
and 48). In each of the 11 failing processes checked, the same op
passed validation once only its quantize params union was zeroed. No unit test is added: the
failure depends on what the stack holds, so a test cannot fail
reliably without the fix.
During decomposition, exir calls a partitioner's ops_to_not_decompose
filter on the nodes of the ATen program one at a time, in graph order.
Each node whose op is in the list and that the filter accepts gets its
target replaced right away by a copy in the EDGE_DO_NOT_DECOMP
namespace (except ops that exir does not preserve, such as mutable and
aliasing ops), and the original target comes back only after that
partitioner's decomposition. So a filter that looks at other nodes, for
example to ask whether an input comes from a dequantize op, sees that
copy for every earlier node that was replaced, not the original op.

The docstring said which program each filter is applied to, but not
this. Partitioner authors now find it there instead of in exir's code.
A constant op such as full can end up as an output of a QNN partition,
either because it is a graph output or because the op that reads it
stays on CPU. QNN then writes it with a Reshape from a static tensor
(see define_constant). Constant ops skip the QNN op check, and HTP
cannot build such a constant with more than 5 dims, so the whole graph
fails to build:

    return x * 2, torch.zeros(1, 2, 1, 1, 3, 4)

    OpDef_Const failed to generate tensor
    Failed to add node to Qnn Graph with error: 6007
    AssertionError: Failed to generate Qnn context binary.

Before constant outputs were written with a Reshape, the same program
built, but it failed when it ran, because QNN never published the
constant.

On HTP, the partitioner now leaves constant ops of more than 5 dims on
CPU. A new test lowers a 6-dim constant that is a graph output, and one
that a CPU op reads, and checks that the multiply is delegated and the
constant is not. Both cases fail without this change.
In HTP float16, Concat can read a static bool input of all True as
False, with no error. For example, with a positive x of shape (2, 3), a
mask built from a computed mask and a constant one comes back wrong on
the x86 simulator of QNN SDK 2.51:

    torch.cat([x > 0, torch.full((2, 2), True)], dim=1)
    # QNN:   [[True, True, True, False, False], ...]
    # eager: [[True, True, True, True, True], ...]

The static tensor ExecuTorch gives QNN holds the right values: QNN
Saver records 0x01 for each element. The wrong result also depends on
the shapes and on where the static input is. At the same shapes,
constants with mixed values, float and int32 cats with a constant, and
quantized graphs come out right.

In HTP float16, the partitioner now keeps on CPU a bool cat that reads
a static tensor: a constant op such as full, or a parameter, a buffer
that the model does not update, or a lifted constant. Since the failing
shapes are not known in advance, this also moves some cats that came
out right, which can split a graph into 2 delegates. A constant op such
as full can then become a partition output, which QNN writes with a
Reshape. The partitioner keeps the fp16 HTP check in one flag,
is_htp_fp16, which the dilated conv VTCM warning now reads too.

A new test checks that, in HTP float16, a bool cat of a full or of a
buffer stays on CPU, that the same float32 cat is delegated, and that a
quantized bool cat stays delegated. Both cat delegate tests gain a bool
cat of a full. The floating-point one comes out wrong without this
change, and the quantized one checks the values of the cat that stays
delegated.
…n I64toI32

I64toI32 keeps the int64 results of ops that QNN has no builder for,
since they run on CPU. But some ops that do have a builder also stay on
CPU, and their int64 results were still cast to int32, which wraps:

    x.sum(-1)    # int32 input [[2**30, 2**30]], sum in skip_node_op_set
    # before: -2147483648, the sum wrapped around in the int32 cast
    # after:  2147483648, as in eager

The same happened to an op the partitioner skips by node name, and to
an op that reads float64, such as the `.long()` in
`(x.double() * x.double()).long()`.

I64toI32 now takes the partitioner's skip_node_id_set and
skip_node_op_set, which the pass manager already hands to passes that
ask for them. Before it rewrites the graph, it marks the ops that stay
on CPU for one of these reasons, using must_stay_on_cpu for the data
types and shapes, and treats them like ops without a builder.

As in RecomposeRmsNorm, a node name in skip_node_id_set is matched
against the graph the pass receives.
…r int64 inputs down

I64toI32 keeps the int64 result of an op that stays on CPU, so a
product, for example, does not wrap. But the inputs of such an op could
still reach it as int32, and then the value wrapped with no error:

- When a CPU op read such a result together with an int32 tensor, the
  pass cast the int64 result down to int32 first, so that ops like
  masked_scatter get matching dtypes:

      x = torch.tensor([100000, 100000, 3, 1])
      torch.fmod(x.prod(0, keepdim=True), x.sum(0, keepdim=True))
      # eager: [12]   before this change: [-169780]

  The same happened to x.prod(-1).index_select(0, index) with int32
  indices when index_select is skipped: 65536 * 65536 came back as 0.
- A CPU op read an int64 graph input or buffer through the int32 cast
  made for QNN ops. x.prod(-1) with x = [[2**33, 3]] returned 0.
- A CPU op read the int32 cast of a QNN result that was int64. With mul
  skipped, s * s * s * s for s = (x > 0).sum(-1) wrapped at 300**4.

A CPU op now reads every value that was int64 in the original graph as
int64. It reads an int64 input itself, not its int32 cast, and it reads
a QNN result that is int32 only because of the pass cast back up, which
loses nothing. Its other inputs keep their dtype. A cast up whose input comes
from QNN joins its partition, like the index casts the pass already
adds, and a cast with nothing else around it stays on CPU.

Each cast between QNN and CPU now takes the meta of the value it casts,
not of the op that first reads it, and the pass drops its internal
QCOM_KEEP_INT64 mark once it is done.

The CPU op test gains the fmod case and an int64 input, the test of ops
kept on CPU gains the skipped index_select and the skipped muls, and a
new test checks the meta of the casts. All fail without this change.
The int64 result test also gains a masked_scatter whose source is k + k,
to check that both inputs still get the same dtype.
ConvertFullLikeToFull turned every full_like of a static shape into a
new full node. A full_like named in skip_node_id_set or skip_node_op_set
then lost its skip, since the new node has another name and op. I64toI32
runs later and keeps int64 only for skipped ops, so it cast the full to
int32 and back, and a large fill wrapped:

    torch.full_like(x, 2**40, dtype=torch.int64)  # x is float32
    # with full_like in skip_node_op_set: 0 instead of 1099511627776

The pass now takes both skip sets, which the pass manager already hands
to every pass that accepts them, and leaves a skipped full_like alone.
It stays on CPU, so the problem the pass solves, QNN dropping the ops
that only feed a delegate output, does not apply to it.

A new test runs the pass and I64toI32 with each skip set and checks that
the full_like stays and that the fill matches eager.
…ts only in I64toI32

_record_original_output_dtype stored a list of dtypes for a graph
output whose value is a tuple, such as a returned topk. Its only reader,
_preserve_output_dtype, skips such nodes, because each of their
getitems is a graph output too and is cast back on its own. So the list
was never read.

The original dtype is now recorded only for tensor outputs, and the
tuple check in _preserve_output_dtype goes away with it. The test that
returns a topk of an int64 sum still matches eager for both outputs.
The ops_to_not_decompose docstring said that exir calls the filter on
the nodes in graph order. exir walks all top-level nodes first and only
then the nodes of each cond, map and scan submodule. So a filter on a
submodule node already sees every replaced top-level node, including
those after the cond. The docstring now says that, and that a filter
sees the EDGE_DO_NOT_DECOMP op for each node already replaced before it.
…mposeRmsNorm

RecomposeRmsNorm erased each rebuilt norm and its dead inputs with its
own helper, _erase_unused. The pass already removes dead code with
graph.eliminate_dead_code() before it starts, so right after a rebuild
the only dead nodes are the ones that rebuild left. Calling
eliminate_dead_code() there does the same thing, and dead nodes are
erased one way. It still runs right away, so a node that another norm
shares has the right users when _is_skipped counts them.

The RMSNorm tests in test_passes.py pass unchanged.
…or on CPU

In HTP float16, the partitioner keeps on CPU a bool cat that reads a
static tensor, because HTP Concat can read such an input of all True as
False. The check looked only at the cat's direct inputs. A static tensor
reached through a reshape was missed, so the cat stayed delegated, and
on the QNN x86 simulator it came back wrong in the same way:

    torch.cat([x > 0, mask.reshape(1, 4)], 1)  # mask is a bool buffer
    # QNN:   [1, 0, 1, 0, 0, 0, 0, 0]
    # eager: [1, 0, 1, 0, 1, 1, 1, 1]

The check now looks through view_copy, squeeze_copy, unsqueeze_copy,
permute_copy and expand_copy, which only change the shape or the order
of the dims, to the tensor they read. A permute or an expand of a static
tensor read right at the shapes tried on the simulator, but whether
Concat misreads depends on the shapes, so those cats stay on CPU too,
like a cat of the static tensor itself.

The partitioner test gains a view and an expand of a full, and a
reshape and a permute of a buffer. All 4 stay delegated without this
change.
On HTP, the partitioner leaves a partition on CPU when it reads no
runtime value. It merges partitions even when no data flows between
them, unless that makes a cycle, and it counts the output of a CPU op as
a runtime value. I64toI32 and RecomposeRmsNorm each guessed this in
their own way, and the guesses were wrong in some graphs:

- I64toI32 cast the int64 result of such an op to int32 on CPU. For
  example torch.full_like(x, 2**40, dtype=torch.int64) became a full
  that HTP left on CPU, and it returned 0 with no error.
- RecomposeRmsNorm did not rebuild a norm of constants that the
  partitioner merges with unrelated ops that read an input, or a norm
  whose input is the output of a CPU op. The partitioner still delegated
  the decomposed norm, and in float16 it overflowed with no error.

The partitioner already estimates this for the ops it keeps whole. That
estimate moves to find_nodes_left_on_cpu in partition/utils.py, and both
passes use it too, with may_be_delegated telling which ops the
partitioner may take. I64toI32 keeps int64 for every op that the
partitioner may not take, also an op outside ATen that has no builder,
and on HTP for every op the estimate leaves on CPU. Ops that the op
packages in the compiler specs build count as ops QNN takes. QNN's op
check is not known this early, so an op it rejects is still cast.

Also:
- capture_program takes skip_node_id_set and skip_node_op_set and
  passes them to the edge passes, like to_edge_transform_and_lower_to_qnn.
- The mutable buffer log says that buffers that are not floating point
  stay on CPU, as the code does.
- The RMSNorm runtime value tests run the export pipeline first, as the
  real lowering does, so pow has a lifted exponent and a QNN builder.
  The buffer update in one of them is a clamp_, because the pipeline
  turns a fill_ into a full that no longer reads the buffer. A norm of
  constants is tested in a graph with no other runtime value, and a
  norm next to a mask index is now expected to be rebuilt, since the
  partitioner delegates it with the comparison.
… the passes

RecomposeRmsNorm and I64toI32 estimate which ops the partitioner leaves
on CPU. They asked may_be_delegated about each op, and it did not know
two rules that QnnOperatorSupport applies on HTP: a constant of more
than 5 dims stays on CPU, and in float16 so does a bool cat of a static
tensor. So the passes could expect such an op on QNN while the
partitioner kept it on CPU. For example, a norm of constants that
reaches the input only through such a cat was rebuilt as an rms_norm.
HTP then dropped its partition, which read no runtime value, and
lowering failed, since rms_norm has no CPU kernel:

    Node aten_rms_norm_default ... was not decomposed or delegated

The two rules move to get_htp_skip_reason in partition/utils.py. Both
QnnOperatorSupport and may_be_delegated call it, so they apply the same
rules. get_backend_info reads the backend, the float16 precision and
the op package ops from the compiler specs, for both passes.

QNN's op check and the limits of older HTP architectures still run only
in the partitioner.

The new test lowers such a model with ConstantFolding off, since it
would fold the norm of constants away, and fails without this change.
… in ConvertFullLikeToFull

ConvertFullLikeToFull gave the new full the dtype of the full_like but
not its device, so the full was made on the default device. With an
input on another device, such as CUDA, x + torch.full_like(x, 0.5) then
failed in the next pass that ran the graph:

    FakeTensorDeviceMismatchError: Expected all tensors to be on the same
    device, but found at least two devices, cuda:0 and cpu!

The full now gets the device too.

A full is also always contiguous, while a full_like takes the dim order
of its input, and the pass kept the meta of the full_like. For a
channels last x, x + torch.full_like(x, 0.5) with the add on CPU then
lowered, but failed at runtime:

    Check failed (tensors_have_same_dim_order(a, b, out))

A full_like that is not contiguous is now kept.
…s in the test lowering stage

ToEdgeTransformAndLower in the backend tester passed its own default
compiler specs to the edge passes, and not the skip sets of its
partitioners. With a QnnPartitioner that skips rms_norm,
RecomposeRmsNorm still built an rms_norm, the partitioner left it on
CPU, and lowering failed, since rms_norm has no CPU kernel. And with a
float16 QnnPartitioner on a stage made for quantized HTP, the passes did
not rebuild the float32 part of a norm, so it overflowed in float16.

The stage now passes the specs of its first QNN partitioner and the skip
sets of all of them, as to_edge_transform_and_lower_to_qnn does.
In float16 mode QNN runs every delegated float32 op in float16. A
timestep or rotary embedding takes sin and cos of large angles, like
t*freqs or positions/timescale, and in float16 these come out badly
wrong, for 2 reasons. float16 cannot hold the angle: its values are 1.0
apart from 1024 to 2048, and a timestep embedding with a 4e-3 period
reaches 1571. And HTP's float16 Sin and Cos lose more on top of that for
large angles. On the QNN SDK's x86 simulator for SM8650, the largest
error against float32 was 1.39 for timestep embedding angles of 1000 to
2000, and 0.50 for rotary embedding angles of 300 to 520 (positions up
to 511). A float32 sin of the float16 angle is still off by 0.95 in the
first case, so a better Sin alone would not fix it.

A new to_edge pass, TagPositionalSinCos, tags such a float32 sin or cos
for CPU fallback, together with the ops that compute its angle, so they
compute in float32 as the model does. The partitioner already leaves
nodes tagged QCOM_FALLBACK_NODE on CPU. The pass acts for HTP in float16
precision only, and needs the compiler specs, which
to_edge_transform_and_lower_to_qnn, transform_for_to_edge_pipeline (used
by export_llama), the QNN recipe and the QNN tester give it. Without
them, as in the deprecated capture_program, it does nothing and logs a
debug message. The HTP pass manager always runs it after every other
pass, last before ResolveDebugHandle, so it sees the graph the
partitioner sees. Like ResolveDebugHandle, it is moved there whatever
dep_table says. It is on by default, logs a warning when it tags ops,
and can be turned off in passes_job, for example to keep the delegated
graph of an older program, whose runtime may not have the CPU kernels
these ops need:

    passes_job = get_qnn_pass_manager_cls().get_capture_program_passes()
    passes_job[TagPositionalSinCos][QCOM_PASS_ACTIVATE_KEY] = False
    to_edge_transform_and_lower_to_qnn(
        module, inputs, specs, passes_job=passes_job
    )

The QNN FP16 recipe takes the same passes_job, one job for every
method. The Qualcomm backend docs describe the CPU kernels this needs
and how to turn the pass off.

It applies only when the angle is built from graph inputs, constants,
integer positions and sizes (like position ids or x.shape[1]) by cheap
elementwise and shape ops: arithmetic, pow, exp, log, casts, views,
slices, expand, permute, cat, the arange and where that a linspace
becomes, and the full and cumsum of float positions like
ones(n).cumsum(0). The sin or cos must also be larger than the graph
inputs, and the integers computed from one, that it and its float ops
read, taken together, as an angle that spreads a few positions or
timesteps over a frequency axis is. Fourier features of a graph input,
like sin(x[..., None] * freqs), qualify too. Parameters, buffers the
graph does not change and integers built only from constants and sizes
(an int input counts as a size), like the arange inside a linspace, do
not count. A buffer the graph changes, directly, through a view or as an
out= argument, counts as an input. So a single timestep still qualifies,
while sin(x * 30) of an image, float or integer, of a cat of inputs or
of a mutable state buffer, is not tagged.

A sin or cos stays delegated when it, or an op of its angle, has more
than 262144 elements, so that no large computation, like Fourier
features of a large image, moves to CPU. The limit holds for each sin or
cos on its own, not for all the tagged ops together. That is 1024
positions times a head size of 256: the largest rotary table of the
Llama, Qwen, Gemma and PaliGemma style models at the 1024 token context
of the Qualcomm LLM examples (Llama 3.2 3B and Qwen3 have 131072, Gemma
3 1B 262144, SmolVLA 7712 to 9312). On a Cortex-A78 core a float32 sin
of that size takes about 2.5 ms. A dynamic size counts with its largest
value, so one without a max is not tagged, with a debug message.

Integer positions built only from graph inputs and constants, like
attention_mask.cumsum(-1), mask.sum(-1) + ones(n).cumsum(0) or arange(n)
< k, move to CPU with the angle, including their casts and comparisons.
This matters because I64toI32 puts an int32 cast after each int64 op,
and QNN gives that op the int32 output: a cast left on CPU behind a
delegated int64 op would read the wrong data. Integers computed from
activations stay delegated, and so does any angle built from them, so
every tagged op reads only graph inputs and constants and no delegate is
split. The walk over integers uses an explicit stack, so a long integer
chain does not reach Python's recursion limit.

A sin or cos of an activation, for example after a linear or a conv,
stays delegated, so no heavy op moves to CPU. So does a sin or cos of a
graph input that the graph only moves or casts: the graph does not
compute that angle, so the caller can keep it small. Quantized nodes,
the quantized mode and other backends are not affected.

A rotary embedding that builds its angle with a matmul, as older
Hugging Face Transformers versions do with inv_freq @ position_ids, is
not covered. Transformers 5.22 builds it with a broadcast multiply,
which is. QNN keeps matmul whole for HTP, and a whole matmul has no CPU
kernel, so it cannot simply stay on CPU. Integer positions spread over
the frequency axis before their cast to float, like
(pos[..., None] * arange(n)).float(), are not covered either: the cast
reads as many elements as the angle has, so it does not count as
positional.

The new pass reads the backend from the compiler specs with
get_backend_info and counts runtime inputs with get_runtime_inputs, as
RecomposeRmsNorm does.

Results on the simulator:
- For the 2 cases above the largest error is now 0.0002, the float16
  rounding of the delegated cat after them, with the same number of
  delegates.
- SmolVLA, whose language model builds rotary angles from position ids
  this way: the largest error of the keys it caches drops from 0.79 to
  0.19, with the same number of delegates. With a fix for its uint8
  camera input cast also applied, the end to end action check drops
  from 0.077 to 0.033.

Tests:
- A delegate test lowers a sinusoidal embedding with angles up to 1571,
  built with linspace inside forward, for a batch of timesteps and for a
  single one, and checks its output.
- Another lowers rotary angles from attention_mask.cumsum(-1) positions
  plus an int64 constant made in forward, up to 500, and checks its
  output, so the integer ops and their int32 casts run on CPU end to
  end. A second case also returns the positions and feeds them to a
  delegated op.
- A test in the backends/qualcomm/tests/rework pass tests checks that
  float16 HTP tags the sin, the cos and their angle of a timestep
  embedding, but not the cat after them, and that quantized HTP tags
  nothing.
- A partitioner test checks that such a sin and its angle go to CPU
  while the cos of a linear stays delegated, in one delegate, and that
  turning the pass off delegates the sin again.
- Unit tests, which do not initialize the QNN backend, run the QNN
  to_edge passes: a linspace timestep angle, an exp(-log(10000) * i)
  frequency, a batch of one with buffer or linspace frequencies, an
  angle from an int64 cumsum, from integer or float offset positions or
  from a count over a mask as large as the angle, and aranges of a
  dynamic size or int input, with or without a frequency axis, are
  tagged with every op they come from, and nothing else is tagged; an
  angle computed from a linear or from integers computed from an
  activation (including a float sum), sin(x * 30) of a float or integer
  graph input or of a cat of inputs, a sin of a graph input or of its
  views, a float16 sin, quantized nodes, quantized HTP and the GPU
  backend are not. Others check that a sin or cos of 262144 elements is
  tagged and one of 262400 is not, also with a dynamic size bounded by
  either, that a small cos sliced from a larger table and a dynamic size
  without a max are not; that a mutable buffer, written in place,
  through a slice or a split, or as an out= argument, counts as an
  input, both in a program from torch.export and in an edge program;
  that an angle behind 1000 chained integer adds is tagged; that the
  pass logs a debug message without compiler specs; and that the QNN
  FP16 recipe turns the pass off with the same passes_job and rejects
  one that is not a dict or that is keyed by method name. Another checks
  that a dep_table with no entry for the pass, like the base one, or one
  that orders a pass after it or after ResolveDebugHandle, still runs it
  last before ResolveDebugHandle.
to_edge_transform_and_lower_to_qnn, the QNN FP16 recipe and the other
QNN entry points run only the active passes listed in passes_job, and
the pass manager moves ResolveDebugHandle (on LPAI, followed by
LpaiPartitionFallbackSupport) to the end. A passes_job that lists only
the passes to change, like

    passes_job = {SomePass: {QCOM_PASS_ACTIVATE_KEY: False}}

leaves ResolveDebugHandle out, and failed there with a bare
"list.remove(x): x not in list" that did not say how to build a valid
job. A passes_job that turns ResolveDebugHandle off failed later with
an AssertionError. An entry that only turns a pass on, like
{QCOM_PASS_ACTIVATE_KEY: True}, failed with a KeyError while building
that pass, and so did an entry with no activation key. An entry that is
not a dict failed with a TypeError, also when convert_linear_to_conv2d
turned that pass on.

Before any pass is built, the pass manager now checks that every key of
passes_job is a pass (a class or a callable like functools.partial, not
a method name), that every entry is a dict with an activation key,
that ResolveDebugHandle (on LPAI, LpaiPartitionFallbackSupport too) is
turned on, and that each active entry has its build arguments. Each
raises a ValueError naming the dict to start from for that backend, for
example:

    passes_job = QnnHtpPassManager.get_capture_program_passes()
    passes_job[SomePass][QCOM_PASS_ACTIVATE_KEY] = False

A passes_job that turns these passes on goes through as before,
including a reduced one that runs only the passes it lists and one with
a functools.partial key. The check reads the terminal passes from
_get_terminal_passes, so LPAI no longer needs its own check, and a
docstring that named the removed _validate_edge_passes now says what
the pass manager does.

Test: a passes_job without ResolveDebugHandle, an empty one, one with it
turned off, one with an entry that only turns a pass on, one with an
entry without the activation key, one with an entry that is a bool, or
one keyed by method name, raises the ValueError naming the pass manager,
also with convert_linear_to_conv2d and through the QNN recipe, and a
reduced one with ResolveDebugHandle still yields exactly its passes,
also with a functools.partial key. On LPAI, the same reduced job raises
the ValueError.
The pass manager test in backends/qualcomm/tests/rework builds the
to_edge passes with exported_program=None. Since ConstantFolding joined
the default passes, building it reads the program, so the test fails on
both HTP and LPAI with:

    AttributeError: 'NoneType' object has no attribute 'run_decompositions'

The test now gives it a small exported program. No CI job runs the rework
tests, which is why this was not caught.

Test: test_pass_manager[htp] and test_pass_manager[lpai] in
tests/rework/passes/test.py fail before this change and pass after it.
@shoumikhin
shoumikhin force-pushed the qnn-fp16-positional-sincos branch from 580d9bc to 70abb65 Compare November 21, 2026 12:49
Copilot AI balanced review requested due to automatic review settings November 21, 2026 12:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@shoumikhin

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @shoumikhin's task in 37m 22s —— View job


Review of the last 3 commits of #24601

No blockers in the last 3 commits; of the 5 minor findings, the main ones are that the 262144-element limit can't be changed through passes_job and silently leaves the longest rotary tables in fp16, and that the recipe checks passes_job only after export.

Detailed review

PR Review: #24601

Summary

The last 3 commits:

  • add TagPositionalSinCos, which keeps positional sin/cos and the ops that build their angle on CPU for HTP fp16;
  • check passes_job before any pass is built;
  • give the rework pass manager test a program.

There are no blockers; five minor issues and two wording notes are below. All 15 changed files were reviewed by reading the code. Python can't run in this job, so graph behavior comes from hand traces and no test was run.

API Design

  • backends/qualcomm/_passes/tag_positional_sin_cos.py:28, :125, :150-160: the size limit can't be changed through passes_job, and nothing visible is logged when it applies.
    • _MAX_NUMEL is a module constant, and __init__(self, edge_program, compiler_specs=None) takes no limit. get_capture_program_passes() takes its kwargs from the __init__ signature (qnn_pass_manager.py:377-390), so it exposes no setting for the limit.
    • When the limit stops a sin or cos from being tagged, the pass logs at debug level only. The skipped cases are the ones fp16 hurts most, since the limit is sized for the 1024-token examples. A rotary table built in forward for 4096 positions × head size 128 has 524288 elements and angles up to 4095, where fp16 values are 2.0 apart. It stays delegated, with no message at the default log level.
    • The debug text also says "A dynamic size needs a max to be tagged." for static sizes that are just too large.
    • Suggest a max_numel=_MAX_NUMEL constructor argument, plus a warning that names the node and says whether its size is too large or unbounded.
    • Fix this →
  • backends/qualcomm/recipes/qnn_recipe_provider.py:139-148: the recipe checks passes_job only after export.
    • create_recipe checks only the container type and string keys. A job without ResolveDebugHandle, or with an entry that only sets QCOM_PASS_ACTIVATE_KEY, fails only when the lowering lambda runs, which is after torch.export. The PR's own test calls recipe.lowering_recipe.edge_transform_passes[0](None, program) to reach the error (test_passes.py:3670-3674).
    • The check also sits in _validate_partitioner_parameters, but passes_job configures the to_edge passes, not the partitioner.
    • Suggest running the checks of _get_active_edge_passes (qnn_pass_manager.py:453-481) from create_recipe, in a validator of their own. None of those checks need the solved pass order.
    • Fix this →

Testing

  • backends/qualcomm/tests/test_passes.py:3651-3661: test_to_edge_passes_need_resolve_debug_handle can't tell which check fired.
    • Every case asserts only the QnnHtpPassManager.get_capture_program_passes suffix, which all _invalid_passes_job errors share.
    • If the callable(p) check (qnn_pass_manager.py:457-460) were deleted, the test would still pass: by_method = {"forward": passes_job} would fail the activation-key check instead, with the same suffix.
    • Asserting each case's own text ("keyed by pass", "is not a dict with the", "must turn on", "has no") ties each check to its case.
    • Fix this →

Code Quality

  • backends/qualcomm/_passes/backends/htp/qnn_htp_pass_manager.py:40-47: the run-last rule matches only the bare class, and it sits in a hook named for terminal passes.
    • A key is moved only if it is TagPositionalSinCos itself. But 8aa01ee accepts any callable key and tests a functools.partial one (test_passes.py:3686-3699).
    • A functools.partial(TagPositionalSinCos) key or a subclass key is not moved. No dep_table has an entry for this pass, so it is sorted with no constraint and can run before I64toI32. The commit rules out that order "whatever dep_table says". No caller wraps this pass today, so the problem is latent.
    • The override also puts a non-terminal pass in _move_terminal_passes_to_end. A hook of its own would say what it does, for example a _get_late_passes() whose passes the base moves just before the terminal passes.
    • Fix this →
  • backends/qualcomm/_passes/tag_positional_sin_cos.py:137: self.runtime_inputs is state set inside call.
    • _broadcasts_inputs and _reads_runtime_input read it implicitly, while positions is passed as an argument.
    • RecomposeRmsNorm (recompose_rms_norm.py:220-221), which the commit says this pass follows, keeps it in a local variable, and so does I64toI32 (i64_to_i32.py:97).
    • Fix this →

Specific Comments

  • backends/qualcomm/_passes/qnn_pass_manager.py:457-460: the error for a string key misleads through to_edge_transform_and_lower_to_qnn.
    • Take a method-keyed passes_job that leaves out a method, such as {"prefill": job} while lowering prefill and decode. ensure_graph_specific_dict (utils.py:442-444) hands that whole dict to every method.
    • The user then reads "passes_job is keyed by pass, not by 'prefill'", although method keys work when every method is listed.
    • Fix this →
  • backends/qualcomm/genai_pipeline/strategies/compilation/default_compiler_adapter.py:231-233: the docstring now lists passes_job as something the recipe accepts. But the recipe rejects the per-graph {graph_name: job} dict this adapter takes (:376), so this adapter's passes_job still can't go through the recipe. Fix this →

Recommendation

Approve. Nothing blocks merging. The two worth doing before users run into them are the limit argument with its warning, and the check at recipe creation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. release notes: qualcomm Changes to the Qualcomm backend delegate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants