Skip to content

[Espressif] Fuse binary ReLU and HardTanh patterns - #25177

Open
rascani wants to merge 1 commit into
codex/espressif-fused-quant-subfrom
codex/espressif-fused-quant-activation-fusion
Open

rascani wants to merge 1 commit into
codex/espressif-fused-quant-subfrom
codex/espressif-fused-quant-activation-fusion

Conversation

@rascani

@rascani rascani commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Fuse single-consumer add, subtract, or multiply followed by ReLU or HardTanh, including ReLU6, into one ESP-NN call. The quantizer observes pattern boundaries so the FQ reference has no intermediate rounding. FoldActivations transfers the final output qparams and clamp bounds before LowerToEsp reuses the producer's shape and numerical eligibility checks.

Forked and separately quantized activations keep their original boundaries, and unsupported producers remain portable. Clamp bounds use the producer's floating dtype to preserve FP16 and BF16 fallback results. The operator schemas, references, and wrappers carry the final clamp bounds, including both the elementwise and broadcast multiply operators introduced by #24579.

Review the quantizer and FoldActivations first, then lowering and the schema/wrapper updates. Shared dialect and S3 cases cover forks, broadcasts, saturation, fallback, asymmetric/positive/negative HardTanh bounds, and endpoint rounding. Explicit output grids exercise narrower clamp bounds for both multiply wrappers.

Validation: the restacked tip passed 402 focused Python tests covering multiply, add, subtraction, binary activation fusion, numerical preparation, and the pass manager. Scoped lint passed. Earlier optimized ESP32-S3 emulator qualification passed all 55 selected multiply/activation cases with ESP-NN 1.5.0, before the Python-only binary preparation cleanup.

Authored with OpenAI Codex.

@pytorch-bot

pytorch-bot Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25177

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 3 Cancelled Jobs, 1 Pending

As of commit 89c0ce8 with merge base 351e938 (image):

NEW FAILURE - The following job has failed:

CANCELLED JOBS - The following jobs were cancelled. Please retry:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@rascani
rascani added this pull request to stack #25158 November 17, 2026 18:10
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@rascani

rascani commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

Please review the current revision (e5cd902e17e0b0d86f06999527cb0372fd655655) against this PR's base branch, codex/espressif-fused-quant-sub. This PR is part of stack #25158. Re-check any previous findings against the updated code and focus on actionable correctness, quantization/numerical, memory-safety, and test-coverage issues.

@claude

claude Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Claude finished @rascani's task in 24m 37s —— View job


Summary: No blockers in e5cd902, but four minor issues are worth fixing: the C++ add/mul wrappers don't check the new clamp bounds, the tests never send narrowed clamp bounds through broadcast mul or the portable fallback, a FoldActivations check can leave an already-grouped pattern unaccelerated, and the esp_nn::quantized_mul_int8.out schema changes in place.

Detailed review

PR Review: #25177

Summary

This PR fuses a single-consumer add, sub or mul followed by ReLU or HardTanh (including ReLU6) into one ESP-NN call:

  • _EspFusedQuantizer removes the observer between the binary op and the activation.
  • FoldActivations moves the activation's output qparams and clamp bounds onto the binary op.
  • activation_min/activation_max are added to the esp_nn add/mul schemas, the Python reference and the C++ wrappers.

I found no blockers and four minor issues, listed below. This PR has no earlier review findings, so there was nothing to re-check. I could not run the tests, because this job isn't allowed to execute Python. The ESP-NN kernel sources are not in the checkout, so I didn't read them.

Code Quality

  • The C++ wrappers pass the program's activation_min/activation_max to ESP-NN without checking them.

    • Where: backends/espressif/ops/op_quantized_add.cpp:69-70 and backends/espressif/ops/op_quantized_mul.cpp:63-64, 76-77.
    • Why it matters: These bounds used to be hard-coded to -128/127. Every other argument in these wrappers goes through ET_KERNEL_CHECK, and the delegate runtime checks the same values with valid_activation_range (backends/espressif/espnn_runtime/src/xts3_graph.cpp:69-71).
    • Trigger: a hand-built or corrupted .pte with, say, activation_max = 200, or with min > max. The values reach ESP-NN unchanged and the int8 output is wrong. The size checks are unchanged, so nothing reads or writes out of bounds. Export already rejects bad bounds (lower_to_esp.py:156), so this is extra safety rather than a live bug.
    • Fix: in both wrappers, next to the existing checks, add ET_KERNEL_CHECK(ctx, -128 <= activation_min && activation_min <= activation_max && activation_max <= 127, InvalidArgument, out);. Fix this →
  • FoldActivations can skip a pair after the quantizer has already removed the observer between them, and then neither op is accelerated.

    • Where: backends/espressif/passes/fold_activations.py:52-69, compared with backends/espressif/quantizer/esp_fused_quantizer.py:150-187.
    • How it happens: the quantizer groups pairs by graph shape alone. FoldActivations then skips pairs on extra conditions. The strictest is that the output range must be exactly (-128, 127) (:57).
    • Result: if a grouped pair is skipped, the binary op has no out qparams and the activation has no inp qparams. LowerToEsp can lower neither, so both run as portable float ops. On the base branch they ran as an ESP-NN add/mul plus a LUT. The numbers are still correct.
    • Reachability: get_esp_fused_quant_quantizer() can't produce this today. A narrow-range output such as (-127, 127) would.
    • Fix: the :57 check isn't needed for exact results. QuantParamsStruct.quantize (exir/dialects/fused_quant/ops.py:215-223) already clamps the bounds into [quant_min, quant_max], and _qparams(..., allow_output_clamp=True) accepts any int8 sub-range. Removing the check closes the main gap. Fix this →

Testing

  • No test sends narrowed clamp bounds through broadcast mul or the portable fallback, and the fallback cases would still pass if nothing were folded. All references are in backends/espressif/test/ops/test_binary_activation_fusion.py.
    • Narrowed bounds never reach broadcast mul:
      • All broadcast and fallback cases (:100-131) use relu, relu6 or hardtanh_asymmetric with calibrated qparams. Their folded bounds come out as (-128, 127), as the comment at :282-283 also notes.
      • The tests that force a narrower output grid, test_fold_uses_final_qparams_and_clamp (:219-245) and test_implementation_explicit_clamp (:279-290), only use equal (257,) shapes.
      • So on the emulator, the changed esp_nn_mul_broadcast_channel_s8 call (op_quantized_mul.cpp:67-79) is only ever given (-128, 127).
      • A portable quantize with narrowed out_quant_min/out_quant_max is only tested by the BF16/FP16 Python test (:248-276).
    • Fallback cases can't detect a missing fold: with accelerated=False, ops_after_transforms (:76-81) and the .pte assertions (:160-165) pass whether or not FoldActivations folded. An unfolded pair becomes a portable binary op plus an activation, with the same numbers, because the quantizer already removed the observer between them.
    • Fix:
      • Add hardtanh_positive/hardtanh_negative versions of the broadcast-mul and fallback cases. Their bounds narrow even with calibration, because the observed range includes 0. Alternatively, apply _set_output_grid to a (3, 17) × (17,) mul and to one fallback shape.
      • For non-fork cases, add the decomposed activation op with count 0 to ops_after_transforms, as EspAddRelu in test_add.py does with exir_ops.edge.aten.relu.default: 0.
  • Two quantizer exclusions have no test: scalar other (esp_fused_quantizer.py:167) and alpha != 1 (:172-176). Add cases such as relu(x + 1.0) and relu(torch.sub(x, y, alpha=2)) next to test_quantization_boundary, asserting that the observer between the two ops is kept. Fix this →

Backward Compatibility

  • esp_nn::quantized_mul_int8.out changes its signature in place.
    • Where: backends/espressif/ops/operators.yaml:13 and backends/espressif/ops/operators.py:17-26.
    • What changes: on main this op takes 7 int arguments. This PR adds two required arguments without renaming the op or adding an overload. (quantized_add_int8 is new in this stack, so it isn't affected.)
    • Effect: operators are looked up by name, so a .pte exported before this change still loads on a runtime built from it. It then fails at execution with InvalidProgram from the generated argument-count check (codegen/gen.py:249).
    • Fix: the backend is marked as a prototype, so the break may be acceptable. If so, say so in the PR description or release note. Otherwise, register the clamped version as a new overload and keep the 7-argument kernel.

Recommendation

Approve. Nothing blocks merging. The most useful follow-ups are the bounds check in the wrappers and a broadcast-mul case with narrowed bounds. The emulator currently tests the new ESP-NN clamp path only with equal shapes.

  • Gather context (diff vs origin/codex/espressif-fused-quant-sub: 1 commit, 14 files)
  • Run independent reviews (3 reviewers, all changed files covered)
  • Consolidate findings
  • Fact-check findings against the code (5 candidates dropped as invalid, already-existing patterns, or out of scope)
  • Post final review

@rascani
rascani force-pushed the codex/espressif-fused-quant-activation-fusion branch from e5cd902 to 7cf3893 Compare November 17, 2026 22:05
@rascani
rascani force-pushed the codex/espressif-fused-quant-activation-fusion branch from 7cf3893 to ff96319 Compare November 17, 2026 23:40
@rascani
rascani force-pushed the codex/espressif-fused-quant-activation-fusion branch from ff96319 to ebcd154 Compare November 18, 2026 00:39
@rascani
rascani force-pushed the codex/espressif-fused-quant-activation-fusion branch from 5fe214a to cc91377 Compare November 18, 2026 20:52
@rascani
rascani requested a review from JakeStevens November 18, 2026 21:30
@rascani
rascani marked this pull request as ready for review November 18, 2026 21:30
@rascani
rascani force-pushed the codex/espressif-fused-quant-activation-fusion branch 2 times, most recently from 7140f96 to 6f4aa7d Compare November 18, 2026 23:56
@rascani
rascani removed this pull request from stack #25158 November 18, 2026 23:59
@rascani
rascani added this pull request to stack #25261 November 19, 2026 00:01
@rascani
rascani removed this pull request from stack #25261 November 19, 2026 00:23
@rascani
rascani force-pushed the codex/espressif-fused-quant-activation-fusion branch from 6f4aa7d to 2597b2d Compare November 19, 2026 00:23
@rascani
rascani added this pull request to stack #25262 November 19, 2026 00:23
@rascani
rascani force-pushed the codex/espressif-fused-quant-activation-fusion branch from 2597b2d to 1f866ec Compare November 19, 2026 00:48
Quantize single-consumer add, subtract and multiply followed by ReLU or
HardTanh, including ReLU6, at their pattern boundaries so the reference has
no intermediate rounding. Fold the activation into Fused Quant output
qparams and clamp bounds before ESP replacement. Reuse existing shape and
arithmetic eligibility checks with the final output scale; unsupported
producers retain portable execution. Forked and separately quantized
activations keep their original boundaries. Clamp bounds use the producer's
floating dtype to preserve BF16 and FP16 portable fallback results.

Read the quantizer and FoldActivations first, then the binary lowering
and schema/wrapper changes. Shared dialect and S3 implementation cases cover
forks, broadcasts, saturation, fallback and narrower clamp grids, including
asymmetric, positive and negative HardTanh bounds and rounding at endpoints.

Authored with OpenAI Codex.

This branch was previously deployed

1 inactive deployment
cadence — 89c0ce87 Deployed Nov 21, 2026 by rascani via hifi-op-test / hifi4 #39859
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant