Skip to content

[Espressif] Upgrade ESP-NN to 1.5.0 - #25149

Open
rascani wants to merge 3 commits into
codex/espressif-fused-quant-avgpoolfrom
codex/espressif-esp-nn-1-4-2
Open

rascani wants to merge 3 commits into
codex/espressif-fused-quant-avgpoolfrom
codex/espressif-esp-nn-1-4-2

Conversation

@rascani

@rascani rascani commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Update the default ESP-NN dependency to 1.5.0, including the optimized S3 AvgPool row-stride fix, faster multiply kernels, and convolution/pooling alignment fixes. Rectangular AvgPool outputs now lower to ESP-NN, removing the temporary shape checks from lowering and the kernel wrapper. ESP-IDF manifests accept compatible releases through ^1.5.0 ranges. A shared host source fetch has one default release tag and an ESPNN_LOCAL_PATH override, without custom version checks. The new vendor scratch APIs are left for later operator work.

Use the vendor dispatcher for large pointwise convolutions and max pooling with channel counts not divisible by four. ESP-NN 1.5.0 chunks the S3 pointwise calls and handles pooling alignment and channel tails, so the delegate no longer forces those cases to ANSI. Explicit reference execution remains available for ordinary and phase-based convolution and max pooling.

The wide/tall operator cases require acceleration in both channel paths, with rectangular outputs also exercised through padding, batches, chaining, and float consumers. Four Conv → AvgPool delegate cases add coverage to the existing test/export suite. Two pointwise cases cover 65,792 output positions with four or eight input channels, alongside the existing six-channel max-pool regression. Seven public-export assertions verify that these vendor regression fixtures still delegate to the expected ESP-NN operators, so portable fallback cannot mask a regression.

The version upgrade was qualified with the stock registry package on the ESP32-S3 emulator: all 76 operator-library cases passed, and all 20 existing delegate programs passed in both optimized and ANSI modes. The dispatcher cleanup additionally passes the three targeted host reference tests and all three programs in both emulator configurations. The 27 delegate export/reference tests pass with the revised source setup and delegation assertions. Component generation, manifest validation, and scoped lintrunner passed. Physical hardware and dual-core execution were not tested.

The version-only size comparison against matching 1.4.2 test firmware with identical model bundles measured app flash changes of +2896 bytes for the operator library, +4512 for the optimized delegate, and -224 for ANSI. Internal RAM changed by 0, +144, and -16 bytes respectively; the optimized increase was RAM-resident code. These measurements precede the dispatcher cleanup. Separately, both S3 multiply assembly stack frames grow from 120 to 336 bytes.

Authored with OpenAI Codex.

@rascani
rascani added this pull request to stack #25135 November 17, 2026 00:40
@pytorch-bot

pytorch-bot Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25149

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 3c3ca5f with merge base 351e938 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@rascani
rascani force-pushed the codex/espressif-esp-nn-1-4-2 branch from 66d6a20 to 791bc08 Compare November 17, 2026 01:10
@rascani rascani changed the title [Espressif] Upgrade ESP-NN to 1.4.2 for rectangular AvgPool [Espressif] Upgrade ESP-NN to 1.5.0 Nov 17, 2026
@rascani
rascani removed this pull request from stack #25135 November 17, 2026 04:00
@rascani
rascani added this pull request to stack #25158 November 17, 2026 04:00
@rascani

rascani commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@claude review

@claude

claude Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Claude finished @rascani's task in 2m 34s —— View job


This PR moves the ESP-NN pin to 1.5.0 and removes the square-output guards on AvgPool2d. I found no blockers, only two minor test gaps and one minor doc gap for projects that bring their own ESP-NN.

Detailed review

PR Review: #25149

Summary

The PR pins all ESP-NN dependencies to 1.5.0 and removes the rectangular-output guards in AvgPool2d lowering and in the kernel wrapper, because 1.5.0 fixes the S3 row-stride bug. The issues are minor: a project that brings its own ESP-NN gets no minimum version, and there are two test coverage gaps.

Infrastructure

  • backends/espressif/passes/fused_quant_to_esp_pass.py:246, backends/espressif/ops/op_quantized_avg_pool2d.cpp:75: these guards are now protected only by the manifest pin.
    • The bypass: a project that doesn't use the component manager can supply its own components/esp-nn. examples/espressif/idf_component_template/README.md:92-95 documents this, and CI tests it in .ci/scripts/test_espressif_idf_component.sh:401-420. That path never sees the 1.5.0 pin.
    • Failure: with a vendored ESP-NN 1.4.1 or older on S3, a rectangular AvgPool now runs on the buggy kernel, where it used to fall back to the portable kernel. Two examples are tall_output_channels_4 (3x2 output) and wide_output_channels_4. The result is silently wrong output. For tall outputs it may also write past the output buffer, since the row stride is output height; this part is suspected and was not run.
    • Fix: add a line next to the components/esp-nn note in that README saying the local copy must be ESP-NN 1.5.0 or newer. A compile-time check would be better if ESP-NN exposes a version macro (I haven't checked whether it does).

Testing

  • backends/espressif/test/test_int8_ops.py:275-278: the four new avgpool-* delegate cases only check the int8 error tolerance.
    • They never check that the lowered graph actually contains AVG_POOL2D, which test_detection_head_uses_copy_and_no_rank5_tensors does for its own ops.
    • If the pool were decomposed or ended up outside the delegate, these host tests would still pass, so they don't prove the rectangular ESP-NN path is the one being exercised.
    • Fix: add an opcode check for these cases.
  • backends/espressif/test/ops/test_avgpool2d.py:70-96: the reshaped cases only produce wide outputs.
    • batches gives 2x3, negative_padding 4x5, identity 3x5, float_users 2x3, and chain 4x5 then 2x2.
    • Tall outputs (height > width) appear only in the plain tall_output_channels_{4,8} cases. Tall is the direction where a wrong row stride overruns the buffer.
    • The PR description says rectangular outputs are covered through padding, batches, chaining and float users, but the tall direction is not. Making one of these cases tall, such as batches or negative_padding, would close the gap.

Recommendation

Approve

Nothing blocks merging. The README line for vendored ESP-NN is the most useful follow-up.

@rascani
rascani requested a review from JakeStevens November 18, 2026 00:46
@rascani
rascani marked this pull request as ready for review November 18, 2026 00:46
@rascani
rascani force-pushed the codex/espressif-esp-nn-1-4-2 branch from 791d8f1 to 29ecdcd Compare November 18, 2026 18:17
@rascani
rascani force-pushed the codex/espressif-esp-nn-1-4-2 branch from 29ecdcd to aeaf8ea Compare November 18, 2026 20:52
@vikramdattu

Copy link
Copy Markdown
Contributor

Hi @rascani, ESP-NN maintainer here. Thanks for keeping the backend current with ESP-NN.

With ESP-NN 1.5.0, two workarounds in the delegate's int8_interpreter.cpp are no longer needed:

  • 1x1 convolution above 65535 output positions. 1.5.0 splits the ESP32-S3 pointwise call into rows of at most 65528 positions (espressif/esp-nn@2865a6f91c), so esp_nn_conv_s8 writes every output.
  • Max pool with channels not a multiple of 4. 1.5.0's esp_nn_max_pool_s8 checks the channel count and buffer alignment itself and falls back to its C path (espressif/esp-nn@1220e3faac), so any channel count is correct.

Both workarounds also sent these shapes to the ANSI kernels on every other chip. With this PR's 1.5.0 pin they can go. The patch below removes them and adds two 1x1 cases over 256 x 257 = 65792 positions, one per ESP32-S3 pointwise path (input channels a multiple of 8, and not). max-pool-six-channels already covers the pool.

Patch on top of this PR
diff --git a/backends/espressif/espnn_runtime/src/int8_interpreter.cpp b/backends/espressif/espnn_runtime/src/int8_interpreter.cpp
index 0481746..227da1c 100644
--- a/backends/espressif/espnn_runtime/src/int8_interpreter.cpp
+++ b/backends/espressif/espnn_runtime/src/int8_interpreter.cpp
@@ -209,15 +209,6 @@ int8_t quantize_value(float value, float inverse_scale, int32_t zero_point) {
       static_cast<int32_t>(std::nearbyintf(scaled)) + zero_point);
 }
 
-// esp-nn's ESP32-S3 pointwise convolution passes the number of output
-// positions as a uint16_t width and leaves the rest unwritten.
-bool pointwise_beyond_uint16(
-    const data_dims_t& filter,
-    const data_dims_t& output) {
-  return filter.width == 1 && filter.height == 1 &&
-      static_cast<int64_t>(output.width) * output.height > UINT16_MAX;
-}
-
 } // namespace
 
 // Fields that no operation uses together share storage: the input list
@@ -1226,7 +1217,7 @@ bool Int8Interpreter::run_operation(uint32_t index) {
         run_generic_conv(s, input, output);
         return true;
       }
-      if (reference || pointwise_beyond_uint16(s.filter_dims, s.output_dims)) {
+      if (reference) {
         esp_nn_conv_s8_ansi(
             &s.input_dims,
             input,
@@ -1380,9 +1371,7 @@ bool Int8Interpreter::run_operation(uint32_t index) {
                      sw = s.pool[3];
       const uint16_t ph = s.pool[4], pw = s.pool[5];
       if (s.opcode == OpCode::MAX_POOL2D) {
-        // The ESP32-S3 kernel takes a multiple of four channels and writes
-        // nothing otherwise.
-        if (reference || channels % 4 != 0) {
+        if (reference) {
           esp_nn_max_pool_s8_ansi(
               input,
               in_w,
@@ -1676,7 +1665,7 @@ void Int8Interpreter::run_dilated_by_phases(
       data_dims_t tile_dims{tile_w, tile_h, in_c, 1};
       data_dims_t result_dims{cols, rows, out_c, 1};
       if (s.opcode == OpCode::CONV2D) {
-        if (reference || pointwise_beyond_uint16(s.filter_dims, result_dims)) {
+        if (reference) {
           esp_nn_conv_s8_ansi(
               &tile_dims,
               tile,
diff --git a/backends/espressif/test/test_int8_ops.py b/backends/espressif/test/test_int8_ops.py
index 7e5c1a9..9935de1 100644
--- a/backends/espressif/test/test_int8_ops.py
+++ b/backends/espressif/test/test_int8_ops.py
@@ -248,6 +248,24 @@ class ConvAvgPool(nn.Module):
         return F.avg_pool2d(self.conv(x), 2)
 
 
+class PointwiseConv(nn.Module):
+    """A 1x1 convolution over 256 x 257 = 65792 positions, past uint16_t.
+
+    Nearest upsampling builds the large map inside the delegate, after a
+    small convolution quantizes it, and the large convolution has one output
+    channel, so the bundled inputs and outputs stay small.
+    """
+
+    def __init__(self, channels):
+        super().__init__()
+        self.stem = nn.Conv2d(channels, channels, 1)
+        self.conv = nn.Conv2d(channels, 1, 1)
+
+    def forward(self, x):
+        y = F.interpolate(self.stem(x), size=(256, 257), mode="nearest")
+        return self.conv(y)
+
+
 class GroupedConvs(nn.Module):
     """Grouped (not depthwise) convolutions: plain, strided and dilated."""
 
@@ -284,6 +302,10 @@ CASES = [
     ("two-inputs-two-outputs", TwoInputTwoOutput(), [(1, 3, 12, 12), (1, 6)], 2),
     ("transpose-select-stack", TransposeSelect(), [(1, 3, 4, 5)], 2),
     ("max-pool-six-channels", OddChannelPool(), [(1, 3, 8, 8)], 2),
+    # One case per ESP32-S3 pointwise path: input channels a multiple of 8,
+    # and not.
+    ("pointwise-65792-positions-c8", PointwiseConv(8), [(1, 8, 8, 257)], 2),
+    ("pointwise-65792-positions-c4", PointwiseConv(4), [(1, 4, 8, 257)], 2),
     ("avgpool-wide-c4", ConvAvgPool(4), [(1, 4, 5, 7)], 2),
     ("avgpool-tall-c4", ConvAvgPool(4), [(1, 4, 7, 5)], 2),
     ("avgpool-wide-c8", ConvAvgPool(8), [(1, 8, 5, 7)], 2),

We ran these three cases on top of this PR with the patch, through the CI script's delegate flow in esp-emulator 1.1.0, optimized kernels:

Case ESP32-S3 ESP32-P4
max-pool-six-channels PASS PASS
pointwise-65792-positions-c8 PASS PASS
pointwise-65792-positions-c4 PASS PASS

All three are fully delegated and match the bundled reference at the exact tolerance.

Pin the delegate, operator-library, and packaged ESP-IDF components to
ESP-NN 1.5.0 for the S3 AvgPool row-stride fix, faster multiply, and
convolution/pooling alignment fixes. Remove the temporary rectangular
AvgPool checks from lowering and the kernel wrapper. Exercise wide and
tall outputs through the shared operator cases and four delegate cases.

All 76 operator-library cases and 20 delegate programs in each of
optimized and ANSI modes pass on the S3 emulator. Generated component
manifest validation and scoped lintrunner pass. Compared with 1.4.2,
matching test firmware grows by 2896 bytes for the operator library and
4512 bytes for the optimized delegate; ANSI shrinks by 224 bytes.
The new S3 multiply frames require 216 additional stack bytes.

Authored with OpenAI Codex.
Let ESP-NN handle large pointwise convolution maps and max pooling with channel counts not divisible by four. Version 1.5.0 chunks large S3 pointwise calls and handles pooling alignment and channel tails, so the delegate no longer needs to force those cases to ANSI. Keep the explicit reference path in ordinary and phase-based convolution and max pooling.

Add pointwise regressions with 65,792 output positions and four or eight input channels. Those cases and the existing six-channel max-pool case pass the host reference tests and the ESP32-S3 emulator in both optimized and ANSI configurations. Scoped lintrunner and git diff --check pass.

Authored with OpenAI Codex.
@rascani

rascani commented Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

cc @CodeKitty-max

@rascani

rascani commented Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

@claude /pr-review recheck findings

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @rascani's task in 30m 3s —— View job


Re-checked the earlier reviews' findings at 3169ddb: none of the five that were still open has been fixed, the new dispatcher commit widens two of them, and docs/source/backends/espressif/espressif-overview.md:31 still gives ESP-NN 1.4.0 as the minimum; nothing blocks the default managed build.

Detailed review

PR Review: #25149

Summary

The PR pins every ESP-NN dependency to 1.5.0 and removes the AvgPool2d square-output guard. The new commit 3169ddb also sends two kinds of operation to ESP-NN's dispatcher instead of ANSI: large pointwise convolutions, and max pools whose channel count isn't a multiple of 4.

Status of the earlier findings:

Earlier finding Status at 3169ddb Section
No ESP-NN ≥ 1.5.0 check for a project's own components/esp-nn Open. Wider: the delegate now needs 1.5.0 too Backward Compatibility
Exact "1.5.0" pins contradict the README's "or newer" Open Infrastructure
Nothing requires the emulator programs to be fully delegated Open. Wider: now also the pointwise and max-pool cases Testing
Tall AvgPool outputs are tested only in their simplest form Open Testing
The AvgPool wrapper doesn't check out geometry (predates the PR) Open Code Quality

There are two new findings: the overview doc is stale, and the pointwise cases can't detect a misplaced chunk.

I didn't run any tests. ESP-NN's sources aren't in the checkout, so what I say below about vendor behaviour comes from the comments this PR removes and from the maintainer's note.

Code Quality

  • backends/espressif/ops/op_quantized_avg_pool2d.cpp:43-83: the wrapper doesn't check out against the pooling geometry. This predates the PR.
    • It checks dims, int32 bounds and nbytes.
    • It doesn't check out.size(0) == batches, out.size(3) == channels, or H/W against (in + 2·pad − k) / stride + 1. Lowering checks all three at passes/lower_to_esp.py:210-221.
    • Trigger: a malformed .pte with a 1x4x4x4 input, kernel and stride 2, and a 1x3x3x4 out.
      • The windows for output row and column 2 lie entirely outside the input, so their element count is zero.
      • ESP-NN's ANSI kernel divides by that count. This comes from my memory of the vendor code, so confidence is low.
      • With the square term gone, non-square malformed shapes reach the kernel as well.
    • Fix: add the lowering check to the ET_KERNEL_CHECK. Fix this →

Infrastructure

  • backends/espressif/espnn_runtime/idf_component.yml:13, examples/espressif/idf_component_template/idf_component.yml.in:28, backends/espressif/idf_component.yml:11: all three pin exactly "1.5.0", but examples/espressif/idf_component_template/README.md:96 says "1.5.0 or newer".
    • The delegate and the published template used to accept ranges (^1.4.0 and ^1.4). Consumers now can't pick up a 1.5.x patch release. Dependency resolution fails if another component in the app needs a different 1.5.x.
    • docs/source/backends/espressif/espressif-overview.md:31 says only the operator library pins an exact version, so only it can cause such conflicts. After this PR, the default delegate path pins one too.
    • Fix: if newer releases are fine, use ^1.5.0. If bit-exactness with the host reference needs exactly 1.5.0, say "exactly" in the README and in the overview. Fix this →

Testing

  • backends/espressif/test/test_int8_ops.py:300-306: nothing requires the programs this PR depends on to be fully delegated.
    • test_v6_lowering_tracks_pytorch_int8 runs build_graph on the whole model and compares against the Python reference. That shows XTS3 can lower the model, not that EspNNPartitioner delegates it.
    • The emulator programs go through to_espnn_edge, and two things relax the check there:
      • examples/espressif/export_espnn_test_programs.py:107-112 switches to a 1% tolerance whenever cpu_ops is non-empty.
      • .ci/scripts/test_espressif.sh:177-181 reads only name, atol and rtol.
    • Trigger: a partitioner change leaves the pool or the large 1x1 convolution on the CPU.
      • Affected cases: max-pool-six-channels, both pointwise-* cases and the four avgpool-* cases.
      • They keep passing on the portable kernels, and none of the 1.5.0 vendor paths this PR relies on runs.
      • The maintainer observed "fully delegated" once, and CI doesn't check it.
    • Fix: add a host test that runs to_espnn_edge on these cases and asserts that cpu_ops is empty. It should also check that the lowered graph contains the expected opcode (AVG_POOL2D, MAX_POOL2D, CONV2D), the way test_detection_head_uses_copy_and_no_rank5_tensors checks for COPY. Fix this →
  • backends/espressif/test/ops/test_avgpool2d.py:101-157: tall outputs are still tested only in their simplest form.
    • Every case this PR reshaped, plus unbatched, produces a wide output:

      Case Output
      batches 2x3
      rectangular 4x8
      negative_padding 4x5
      identity 3x5
      chain 4x5, then 2x2
      float_users 2x3
      unbatched 2x3
    • The only tall outputs are 3x2, with kernel 2, no padding and batch 1. They come from tall_output_channels_{4,8} and the delegate's avgpool-tall-* cases.

    • Tall is the direction in which 1.4.1's row stride wrote past out.

    • These newly accepted shapes have no test:

      • tall with padding;
      • tall with batch > 1;
      • single-column outputs (out_w == 1 < out_h), which the old guard rejected.
    • Fix: add new cases, keeping the existing ones:

      • the negative_padding parameters on (1, 4, 9, 7), giving 5x4;
      • EspAvgPool2d() on (3, 4, 9, 7), giving 3x2;
      • kernel_size=2 on (1, 4, 7, 2), giving 3x1.

      Fix this →

  • backends/espressif/test/test_int8_ops.py:251-262 (new finding): the pointwise-65792-positions-* cases show that every output gets written, but not that each one lands in the right place.
    • The large convolution has one output channel. A chunk whose output offset leaves out the output-channel stride would still give the right result. Real 1x1 convolutions at this size have many output channels.
    • Nearest upsampling from 8 to 256 rows repeats each source row 32 times. An input offset that is off by whole rows within one of those bands would also go unnoticed.
    • The cases do catch the old truncation, which left 65,536 of the 65,792 outputs unwritten. They don't test ESP-NN's new split, which this PR now relies on.
    • Fix: use bilinear upsampling so that the rows differ; this doesn't change the bundle size. Two output channels would also test the stride, but they double the bundled float output to about 0.5 MB per case. Fix this →

Backward Compatibility

  • docs/source/backends/espressif/espressif-overview.md:31, examples/espressif/idf_component_template/CMakeLists.txt:40-45: ESP-NN older than 1.5.0 is still documented as supported, and nothing rejects it.
    • The overview still says "version 1.4.0 or newer". This part is a new finding.

    • The template accepts a project's own components/esp-nn without checking its version. The sentence at the template's README.md:96 is the only guard.

    • The CI job for this path (.ci/scripts/test_espressif_idf_component.sh:401-420) copies in the managed 1.5.0 package, so it can't catch a stale copy.

    • 3169ddb makes the delegate depend on 1.5.0 too, not just the operator library's AvgPool (espnn_runtime/src/int8_interpreter.cpp:1220, :1374, :1668).

    • Consider an optimized S3 build with a local copy older than 1.5.0. According to the comments this PR removes:

      • pointwise convolutions with more than 65,535 output positions leave outputs unwritten;
      • max pools whose channel count isn't a multiple of 4 write nothing;
      • operator-library AvgPool produces wrong wide outputs. For tall outputs it writes (out_h − 1)·(out_h − out_w)·C bytes past out, which is 8 bytes for tall_output_channels_4.

      Before this PR, all three ran correctly on 1.4.1, through the ANSI or portable fallback.

    • Fix: update the overview line in this PR, since this PR is what makes it wrong. As a follow-up, make configuration fail in the template's else() branch when the local copy is older than 1.5.0. One way is to read version: from its idf_component.yml. Fix this →

Recommendation

Approve

Nothing blocks the default managed build, which can only resolve 1.5.0. Update espressif-overview.md:31 before merging.

The configure-time version check is still the most useful follow-up. Without it, a stale local ESP-NN turns the removed guards into silently wrong results, and into memory corruption for tall AvgPool outputs.

Let ESP-IDF resolve compatible releases through the component manifests. Move the delegate's host source fetch into a shared CMake file with one default pin and ESPNN_LOCAL_PATH override, without parsing vendor version metadata.

Check that the delegate's vendor regression fixtures still lower to ESP-NN, so portable fallback cannot hide a regression.

Authored with OpenAI Codex.

This branch was previously deployed

1 inactive deployment
cadence — 3c3ca5f0 Deployed Nov 21, 2026 by rascani via hifi-op-test / hifi4 #39851
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants