Skip to content

Replace in-place writes before XNNPACK quantizer annotation - #24499

Open
msluszniak wants to merge 3 commits into
pytorch:mainfrom
msluszniak:ms/quantizer-in-place-writes
Open

msluszniak wants to merge 3 commits into
pytorch:mainfrom
msluszniak:ms/quantizer-in-place-writes

Conversation

@msluszniak

@msluszniak msluszniak commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

prepare_pt2e observes a tensor before a later in-place write through a view, so the quantized model ignores what was written (pytorch/ao#5040, e.g. torchvision's batch_images). Per review in pytorch/ao#5046 this belongs in the quantizer: XNNPACKQuantizer now replaces such writes with functional ops in transform_for_annotation. Only writes through select/slice/diagonal views that stay equivalent are rewritten; writes to inputs and buffers, and the other cases listed in the pass docstring, are kept. The QNN wiring is in #25202.

Test plan

New test_replace_inplace_ops.py and test_new_tensor_filled_in_place_through_view (both fail on main). The xnnpack quantizer and transforms suites fail the same tests as main.

cc @kimishpatel @jerryzh168 @metascroy @JakeStevens @nil-is-all

@pytorch-bot

pytorch-bot Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24499

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 1a3b944 with merge base 5c81773 (image):

NEW FAILURE - The following job has failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 3, 2026
@executorch-triage executorch-triage Bot added the community: contribution PRs coming from community (excluding hardware partners) label Nov 3, 2026
@msluszniak

Copy link
Copy Markdown
Contributor Author

@pytorchbot label "release notes: quantization"

@pytorch-bot pytorch-bot Bot added the release notes: quantization Changes to quantization label Nov 3, 2026
@nil-is-all nil-is-all added the module: quantization Issues related to quantization label Nov 9, 2026
@metascroy

Copy link
Copy Markdown
Contributor

@msluszniak can you split this into two PRs: one for XNNPACK quantizer and one for QNN quantizer.

For XNNPACK, tag @JakeStevens for review. For QNN, tag @psiddh for review.

@JakeStevens

Copy link
Copy Markdown
Contributor

@claude review effort=max

@claude

claude Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Reviewing PR #24499

  • Gather context (diff, CLAUDE.md, pr-review skill)
  • Independent review by a panel of three reviewers (new pass, quantizer integration, tests/build files)
  • Look into the periodic CI failures (mv3, vit)
  • Consolidate and fact-check findings
  • Post the final review

Note: Python and git fetch aren't available in this runner, so the review is based on reading code, not on running tests.

View job run

prepare_pt2e observes a tensor before a later in-place write through a view,
so the quantized model ignores what was written (torchvision's batch_images
fills new_full with copy_). Add a replace_in_place_ops transform that rewrites
such writes to their functional variants, and call it from
XNNPACKQuantizer.transform_for_annotation. Writes to graph inputs and buffers
are kept.
@msluszniak

Copy link
Copy Markdown
Contributor Author

@JakeStevens claude review didn't finished. Please re-trigger it.

@msluszniak
msluszniak force-pushed the ms/quantizer-in-place-writes branch from 8c79942 to e3c44dc Compare November 17, 2026 12:31
@JakeStevens

Copy link
Copy Markdown
Contributor

@claude review effort=max

@claude

claude Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Claude finished @JakeStevens's task in 49m 43s —— View job


PR Review: #24499

Request changes: the new replace_in_place_ops pass silently computes wrong fp32 values when a view it has already rewritten is used again (for example, row = buf[0]; row.copy_(x); row.mul_(2) leaves buf[0] == x instead of 2 * x), and it runs for every XNNPACKQuantizer model.

Detailed review

Summary

This PR adds replace_in_place_ops, which rewrites in-place ops on tensors created inside the graph into functional ops. Writes through select/slice/diagonal/as_strided views also get *_scatter ops. The pass runs in XNNPACKQuantizer.transform_for_annotation so that observers see values written through views (pytorch/ao#5040).

  • Problems: one blocker (A) and one major bug (B). Both silently change fp32 results of graphs that are correct on main.
  • How this was checked: three independent reviewers each reported A–H, and I traced A and B by hand. All six changed files were covered.
  • Limits: nothing was executed. This runner can't run Python, and the torch/torchao sources aren't on disk, so items marked suspected depend on torch internals.

Code Quality

A. Blocker: a second write through a rewritten view is lost (backends/transforms/replace_in_place_ops.py:92-108)

After a write is rewritten, later users are redirected:

  • users of the in-place node and of node.args[0] go to the functional value (copy, add, ...);
  • users of each intermediate parent view go to its scatter.

None of these alias the new base. A later in-place op through the same view therefore works on a detached tensor, and so does a later read of the view after another write to the base. The alias guard at :81-87 can't catch this, because it skips views and only looks at aliases created before node.

buf = x.new_zeros(2, 3)
row = buf.select(0, 0)
row.copy_(x)
row.mul_(2)
return buf + 0  # eager: row 0 == 2 * x; after the pass: row 0 == x
  • Graph after the pass: copy(select, x) → select_scatter(new_zeros, copy, 0, 0) → add(select_scatter, 0), and mul(copy, 2) is dead.
  • Other triggers:
    • row += x; row *= 2
    • buf[0].copy_(x).add_(1)
    • row.copy_(x); buf.add_(1); return row * 2, which returns 2 * x instead of 2 * (x + 1)
  • Proxy handling doesn't matter: this happens however make_fx sets proxies after the first write. If the second write's arg0 is the original select, line 94 sends it to the same copy node.
  • Regression: the pass runs inside prepare_pt2e, so the prepared and converted models compute wrong values even where nothing is quantized. These graphs are correct on main. This hits every XNNPACKQuantizer user, and EspNNQuantizer too, since it calls super().transform_for_annotation (backends/espressif/quantizer/espnn_quantizer.py:239).
  • Fix: for later users, re-emit each view on top of the new parent (for example select(select_scatter, 0, 0)), as functionalization does. Later writes then scatter back into the base, and later alias checks still see the alias. A conservative alternative is to skip the rewrite when node, or any node in views, has a user after node. Fix this →

B. Major: aliases from unbind/split/chunk escape the guard (replace_in_place_ops.py:73-87)

The walk at :78 only follows users that are OpOverloads with alias info on arg0. It never follows the operator.getitem outputs of multi-output views (unbind, split, split_with_sizes, chunk, tensor_split). Those getitems sit right after their producer, before the write, so the check at :81-87 never fires.

  • View write: buf = x.new_zeros(2, 3); first, _ = buf.unbind(0); buf.select(0, 0).copy_(x); return first * 2 returns zeros after the pass instead of 2 * x.
  • Direct write: in a, b = y.unbind(0); y.relu_(); return a + b, the sum reads y from before the relu_.
  • Fix: treat getitem users of a node whose schema return has alias info (Tensor(a)[]) as aliases. Fix this →

C. Minor: _get_functional_op maps any foo_ to foo by name (replace_in_place_ops.py:24-31)

  • Metadata-only ops: ops tagged torch.Tag.inplace_view (unsqueeze_, squeeze_, t_, transpose_, as_strided_) only change metadata, but the pass treats them as data writes and scatters them back into the base.
    • buf.select(0, 0).unsqueeze_(0) becomes select_scatter(buf, unsqueeze(select, 0), 0, 0). That puts a [1, 3] source into a [3] slot and fails during calibration.
    • On a square slice, t_ silently writes transposed data into buf.
    • The pass logic is confirmed; whether export keeps these ops on views is suspected.
  • Dtype changes: in-place ops keep self's dtype, but their functional variants don't. Functionalization casts back with _to_copy; this pass doesn't.
    • y.gt_(0) becomes gt, which returns bool, so y.gt_(0); return self.conv(y) feeds a bool tensor to conv. The same applies to eq_, ne_, lt_, logical_and_ and similar ops.
    • An fp16 h.add_(t_fp32) becomes an fp32 add.
  • Fix: skip inplace_view ops. When the functional result's dtype differs, cast back with aten._to_copy or skip the op. An explicit allowlist like Arm's OP_MAP (backends/arm/_passes/rewrite_inplace_arithmetic_pass.py:14-23) avoids both problems by construction, and drops the dynamic getattr lookup that CLAUDE.md discourages. Fix this →

D. Minor: as_strided can only be scattered into the base (replace_in_place_ops.py:20)

as_strided sizes, strides and storage_offset address the underlying storage, but as_strided_scatter(parent, ...) applies them to a clone of parent. The two agree only when parent is the base.

  • Example: in eager, buf = x.new_zeros(4, 4); buf[:, 1].as_strided((2,), (4,)).copy_(y) writes buf[0, 1] and buf[1, 1]. After the pass, the write reaches past the end of a 4-element clone (suspected; not run).
  • Fix: only accept as_strided when view.args[0] is base, or drop the entry. Fix this →

E. Minor: the new nodes carry stale metadata (replace_in_place_ops.py:91, :103)

  • The functional node inherits the in-place node's val. For a write through a view, that is the view's FakeTensor: the view's strides and storage offset, plus the wrong dtype in the C cases.
  • Each scatter copies all of node.meta and only overrides val with the parent's. For an intermediate scatter, the parent is itself a view.
  • dict(node.meta) is a shallow copy, so every new node shares the in-place node's nested nn_module_stack and custom dicts.
  • Fix: compute val for each new node under node.meta["val"].fake_mode, as _convert_scalars_to_attrs uses it (backends/xnnpack/quantizer/xnnpack_quantizer_utils.py:1242-1249), and copy only the provenance keys. Fix this →

Backward Compatibility

F. Direct in-place ops are rewritten too, although they don't have the bug (backends/xnnpack/quantizer/xnnpack_quantizer.py:386)

  • Not needed: prepare_pt2e observes a tensor before an in-place write through a view, so the quantized model ignores its input ao#5040 needs a write through a view. For a direct in-place op, tracing already points later reads at the in-place node, so the observer sees the written value.
  • Still rewritten: the pass rewrites every in-place op on an op output, including ReLU(inplace=True), hardtanh_ and +=. That exposes models that work on main to B, C and E for no gain. test_direct_in_place_op_becomes_functional pins this behaviour on a case that already works on main.
  • What gets quantized changes: _convert_scalars_to_attrs only matches add.Tensor/mul.Tensor (xnnpack_quantizer_utils.py:1221-1224). On main, a scalar y.mul_(0.5) or y.add_(1.0) stays a scalar. After this PR, the scalar is lifted to a _tensor_constant_* buffer and quantized. No test covers this.
  • Fix: limit the rewrite to what ao#5040 needs: writes through views, plus direct writes that a later view write depends on. Or keep the broader scope on purpose, say so in the PR description, and add tests for conv → relu_ and scalar add_/mul_. Fix this →

API Design

G. The docstring overstates what is fixed (replace_in_place_ops.py:41-51)

These writes are left in place without any diagnostic, and they still hit ao#5040:

  • writes through views outside the four-op table, such as view, permute, transpose, unsqueeze or expand (e.g. x.new_zeros(6).view(2, 3)[0].copy_(y));
  • writes through unbind/split getitems, and writes to getitem outputs;
  • writes skipped by the alias guard at :81-87;
  • writes to graph inputs and to get_attr parameters, buffers and constants. With self.buf[0].copy_(x); return self.conv(self.buf), the observer still reads the buffer before the write.

A caller such as the planned QNN follow-up can't tell any of this from the docstring.

Fix: list the supported views and the cases that are kept, and say get_attr rather than "buffers". Log when a write is kept, as exir/inplace_aliasing.py does with logger.debug. Fix this →

H. The new shared pass doesn't follow the backends/transforms/ conventions (replace_in_place_ops.py:40)

  • Pass shape: pass entry points in backends/transforms/ are ExportPass/PassBase subclasses that return PassResult. Even addmm_mm_to_linear.py wraps its functions in AddmmToLinearTransform(ExportPass) (:197).
  • Pipelines: the QNN and Arm annotation pipelines compose pass instances (backends/qualcomm/_passes/qnn_pass_manager.py:454, backends/arm/_passes/arm_pass_manager.py:838), so the QNN follow-up would need a wrapper.
  • Naming: the repo spells it "inplace" (exir/inplace_aliasing.py, exir/passes/reinplace.py, RewriteInplaceArithmeticPass), which suggests a name like ReplaceInplaceOpsPass. Fix this →

Testing

  • backends/xnnpack/test/quantizer/test_pt2e_quantization.py:745: the only assertion is SQNR > 20. That also passes if conv is left in fp32, where SQNR is infinite. Use checkGraphModuleNodes, as at :114-125, to assert that copy_ is gone after prepare_pt2e and that the q/dq nodes are there after convert_pt2e.
  • backends/transforms/test/test_replace_in_place_ops.py:58-71: row is row 1 and the write goes to row 0. Even with the guard removed, the rewrite gives the right numbers and assert_close passes. For the same reason, the comment at :68-69 is wrong. Read an alias that overlaps the write instead (row = buf.select(0, 0)).
  • test_write_to_input_is_kept (:48-56): it never runs gm, so it doesn't check that x is still mutated.
  • Not covered:
    • repeated writes through one view (A);
    • getitem aliases (B);
    • writes to registered buffers being kept, which the docstring promises;
    • the diagonal/as_strided mappings;
    • several writes into one tensor in a loop, as in batch_images;
    • non-strict export: _export at :19 hardcodes strict=True, but non-strict is the torch.export default.

Fix this →

Specific Comments

  • backends/xnnpack/quantizer/xnnpack_quantizer.py:385: the docstring "Transforms scalar values to tensor attributes" is now stale.
  • test_pt2e_quantization.py:723: the name test_buffer_filled_in_place_through_view says "buffer", but buf is a graph-local new_zeros. Registered buffers are exactly what the pass leaves alone. Something like test_new_tensor_filled_in_place_through_view would be clearer.
  • replace_in_place_ops.py:90-104: new_value holds the functional result and then each scatter in turn, which makes the replacement list hard to follow.
  • PR description: after the split, the diff is XNNPACK-only, but the description still mentions QnnQuantizer, the Qualcomm test_passes.py run, and "one test per quantizer". The title also drops the "XNNPACK" that the commit subject has.

Recommendation

Request Changes

  • What blocks approval: A silently changes fp32 results of models that are correct on main, and the pass runs for every XNNPACKQuantizer and EspNNQuantizer user. B fails the same way and should be fixed together with A.
  • Scope: narrowing the pass (F) would keep models without the ao#5040 pattern away from B, C and E.
  • Design question (low confidence): PyTorch functionalization already handles everything A–E get wrong: view replay, multi-output views, inplace_view ops and dtype cast-back. Before hardening a hand-written subset, check whether the graph can be functionalized before annotation. I couldn't check this without running Python.
Progress
  • Gather context (diff, CLAUDE.md, pr-review skill)
  • Independent review by a panel of three reviewers
  • Check CI status: 15 runs passed, 0 failed, 5 still running
  • Consolidate and fact-check findings
  • Post final review

The shallow checkout shares no history with origin/main, so the diff was taken against the PR commit's parent (5c81773, the merge base Dr. CI reports).

A write was rewritten even when the written view, or the in-place op's
result, was used again afterwards, so a second write or read through it
saw the old value. The alias check also missed unbind/split getitems. Keep
those writes, and leave direct in-place ops alone, since later reads
already use the in-place node.

Rewrite only copy_, fill_, zero_ and arithmetic writes through select,
slice and diagonal, compute val for the new nodes under the fake mode, and
copy only provenance meta. Make the transform a PassBase,
ReplaceInplaceOpsPass, and log the writes it keeps.
@msluszniak

Copy link
Copy Markdown
Contributor Author

@JakeStevens addressed the bot review in 3bea937:

  • A, B fixed: a write is now kept when the view, its in-place result, or any alias (including unbind/split getitems) is read after it.
  • C, E, F, G, H fixed: explicit op map, only writes through select/slice/diagonal are rewritten (direct relu_/mul_ untouched), fresh val and provenance-only meta, kept writes logged, now ReplaceInplaceOpsPass. Each has a regression test.
  • D not reproduced (as_strided_scatter matched eager); dropped as_strided anyway.
  • Functionalizing (re-export + run_decompositions({})) fixes these too, but rewrites train-mode batch_norm (breaks test_allow_exported_model_train_eval), can't run on scan/while_loop bodies, and drops dynamic shape ranges. OK to keep the narrow pass?

@msluszniak msluszniak changed the title Replace in-place writes before quantizer annotation Replace in-place writes before XNNPACK quantizer annotation Nov 18, 2026
@msluszniak

Copy link
Copy Markdown
Contributor Author

@metascroy split done: this PR is XNNPACK only (@JakeStevens), and the QNN part is #25202 (@psiddh), which depends on it.

@meta-codesync

meta-codesync Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

@JakeStevens has imported this pull request. If you are a Meta employee, you can view this in D132899587.

@JakeStevens

Copy link
Copy Markdown
Contributor

The BUCK changes fail internally. The issue is that the pytorch testing import brings in a bunch more dependencies that are not set up in the build. We don't actually need to use that. instead, can you please:

  1. Replace the common_utils import with:

    from parameterized import parameterized
  2. Replace both occurrences of:

    @parametrize("strict", [True, False])

    with:

    @parameterized.expand([(True,), (False,)])

    Keep the methods’ strict arguments unchanged.

  3. Delete:

    instantiate_parametrized_tests(TestReplaceInplaceOps)
  4. Add this dependency to test_replace_inplace_ops in backends/transforms/targets.bzl:

    "fbsource//third-party/pypi/parameterized:parameterized",

    That spelling is already used in the same file. Buck needs the declared dependency, not a local pip install.

ExecuTorch’s backends/xnnpack/test/ops/test_gelu.py uses this exact unittest.TestCase + parameterized.expand pattern.

torch.testing's common_utils pulls in dependencies the internal Buck build
lacks.
@msluszniak

Copy link
Copy Markdown
Contributor Author

@JakeStevens done in 1a3b944.

@JakeStevens

Copy link
Copy Markdown
Contributor

ok internal CI + my checks there look good now too, running claude one last time

@JakeStevens

Copy link
Copy Markdown
Contributor

@claude review

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @JakeStevens's task in 6m 13s —— View job


PR Review: #24499

Approve: the earlier blockers (A, B) and the other follow-ups are fixed, and none of the three reviewers found a new correctness bug. What's left is four minor points: one narrow quantization side effect, writes that are still skipped without any visible message, a few gaps in the docstring, and some missing tests.

Detailed review

Summary

This PR adds ReplaceInplaceOpsPass (backends/transforms/replace_inplace_ops.py). The pass rewrites in-place writes through select/slice/diagonal views of tensors created in the graph into functional ops plus *_scatter. XNNPACKQuantizer.transform_for_annotation runs it before _convert_scalars_to_attrs.

Three independent reviewers covered all six changed files, and I re-read the cited code myself. Nothing was executed, because Python isn't available in this runner. Every case the reviewers traced either rewrites correctly or keeps the original write: the scatter order, the user > node redirect, the transitive alias walk (including getitems), and dtype/shape rejection. That holds for the latest version, which replaced torch's common_utils with parameterized (1a3b944).

Code Quality

  • Scalar mul_/add_ through a view now gets quantized (replace_inplace_ops.py:37-40, xnnpack_quantizer.py:387). All three reviewers reported this.

    • What happens: export records buf.select(0, 0).mul_(0.5) as mul_.Tensor with a Python scalar, as test_direct_in_place_scalar_op_is_kept shows for the direct case. The pass turns it into mul.Tensor(select, 0.5). _convert_scalars_to_attrs (xnnpack_quantizer_utils.py:1219-1249) then lifts 0.5 into a _tensor_constant_*, and the mul/add annotator quantizes it.
    • Why it matters: test_direct_in_place_scalar_op_is_kept names this exact outcome as the reason direct ops are left alone, yet it happens on the view path.
    • Fix: when other isn't a Node, emit the .Scalar overloads (aten.mul.Scalar, aten.add.Scalar, ...), which _convert_scalars_to_attrs doesn't match. You could also skip those writes. If lifting is fine here, change the test comment instead.
    • Confidence: low on how much this matters in practice, since nothing was run. Fix this →
  • Skipped writes are still silent (replace_inplace_ops.py:183).

    • What's skipped: some writes that hit the ao#5040 pattern are still left unchanged. Examples are row.copy_(x); row.mul_(2), a base created by an in-place op such as torch.empty(...).fill_(0), and writes through view/permute.
    • Why it matters: these are logged only at debug level, so the user still gets a quantized model that ignores the write, with no visible sign of it.
    • Fix: log at warning when base is a call_function that isn't an input or get_attr, which is the case this pass exists for. Keep debug for KV-cache-style writes into get_attr, so those don't get noisy.
  • New nodes lose custom and from_node (replace_inplace_ops.py:43, :99). _PROVENANCE_KEYS doesn't include them.

    • custom holds PT2E numeric-debug handles and user annotations. The new copy/*_scatter nodes lose both, so the numeric debugger can't trace them back to the original copy_.
    • Copying custom and from_node too keeps provenance without bringing back a stale val.

API Design

  • The docstring's list of skipped writes is incomplete (replace_inplace_ops.py:157-166). It is missing these cases:
    • Base made by an in-place op or a view: in torch.empty(...).fill_(0) or torch.zeros(n * m).view(n, m), the base is the fill_/view node, so _is_alias(base) at :114 skips the write. The reason it logs ("input, get_attr or view") doesn't fit the fill_ case.
    • Control-flow subgraphs: writes inside cond/while_loop/scan bodies are never visited, because only the top-level graph is walked (:174). Your argument against full functionalization cites these bodies, so it's worth stating that this pass doesn't handle them either.
    • Shape changes: the check at :132 also skips writes whose functional op changes the shape, but the last bullet only says "dtype".

Testing

  • Untested op mappings: fill_, zero_, sub_ and div_ are in _INPLACE_TO_FUNCTIONAL but never run.
  • Untested dtype/shape check: the rejection at replace_inplace_ops.py:128-134 has no test. An example would be an fp16 view with add_ of an fp32 tensor, which must stay in place.
  • test_second_write_through_the_same_view_is_kept (test_replace_inplace_ops.py:92) doesn't check what its name says: it only compares against eager and never asserts result.modified is False. A wrong rewrite would only be caught if the numbers happened to differ.
  • Strict export only: the skipped-write tests and test_write_through_diagonal don't run with strict=False, which is the torch.export default.

Recommendation

Approve

Nothing blocks merging. The scalar-lifting point is the one I'd fix before landing, or explicitly accept. The rest can be follow-ups.

Progress
  • Gather context
  • Independent review by a panel of three reviewers
  • Consolidate and fact-check findings
  • Post final review

The diff was taken against 5c81773a, the PR's parent commit and the merge base Dr. CI reports, because this shallow checkout shares no history with origin/main.

@JakeStevens

Copy link
Copy Markdown
Contributor

@msluszniak is the quant thru a view an issue for your usecase?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. community: contribution PRs coming from community (excluding hardware partners) module: quantization Issues related to quantization release notes: quantization Changes to quantization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants