Skip to content

[Espressif] Lower Fused Quant Conv2d with planned scratch - #25154

Open
rascani wants to merge 4 commits into
codex/espressif-scratch-bindingsfrom
codex/espressif-fused-quant-conv2d
Open

rascani wants to merge 4 commits into
codex/espressif-scratch-bindingsfrom
codex/espressif-fused-quant-conv2d

Conversation

@rascani

@rascani rascani commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Lower supported Fused Quant Conv2d calls to ESP-NN using the scratch-size API from #25157. Export inserts an explicit byte tensor whose storage and 16-byte-aligned offset are assigned by ordinary memory planning; the device checks buffer capacity before calling the selected kernel. The application supplies an appropriately aligned arena and serializes ESP-NN execution because the vendor scratch pointer is global.

Support covers static 4-D int8 inputs, constant per-output-channel symmetric int8 weights, optional derived int32 bias, groups=1, and unit dilation, including batches, rectangular kernels, stride, and padding. Unsupported configurations or unavailable sizing bindings retain portable convolution. The recipe and pass manager accept EspTarget and EspKernelMode, including zero-byte scratch for generic or ANSI kernels. The tester defaults to ESPRESSIF_TARGET; explicit target overrides apply to both scratch planning and firmware execution.

Numerical preparation rejects reductions too long for its float32 error bound. Tests assert lowering before execution and check serialized scratch capacity, alignment, and non-overlap. The runner removes stale emulator output before export/build, and CI runs Python operator tests after building the sizing bindings.

Review lowering and allocation first, then the wrapper and shared dialect/implementation cases. Validation: Conv2d host tests pass, including selected-target scratch planning and S3 binding-fallback checks; the staged batch, staged no-bias, stride-remainder, and undersized-scratch cases passed on the optimized ESP32-S3 emulator. Scoped lint passed. Other chip profiles have host sizing/export coverage only.

Authored with OpenAI Codex.

@rascani
rascani added this pull request to stack #25135 November 17, 2026 02:24
@pytorch-bot

pytorch-bot Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25154

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 1 Pending

As of commit f0ccd79 with merge base 351e938 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@rascani
rascani removed this pull request from stack #25135 November 17, 2026 04:00
@rascani
rascani force-pushed the codex/espressif-fused-quant-conv2d branch from c36b1f3 to 967de28 Compare November 17, 2026 04:00
@rascani
rascani changed the base branch from codex/espressif-fused-quant-linear to codex/espressif-scratch-bindings November 17, 2026 04:00
@rascani
rascani added this pull request to stack #25158 November 17, 2026 04:00
@claude

claude Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Claude finished @rascani's task in 47m 15s —— View job


Re-review of 5ae7977: the earlier CI-coverage, stale-log and float32-overflow findings are fixed and nothing blocks merge. Two problems remain: a scratch-bound mismatch can still abort export instead of falling back, and the device kernel still lacks geometry, alignment and shared-lock safeguards.

Detailed review

PR Review: #25154

Summary

This PR lowers fused_quant.convolution to a new esp_nn::quantized_conv2d_int8 operator-library kernel. The kernel uses an int32 derived bias. Its scratch is a memory.alloc tensor sized at export by the host ESP-NN query for the chosen EspTarget/EspKernelMode.

Previous findings, re-checked against this revision

  • Fixed:
    • CI now runs the conv dialect and recipe tests with the bindings required (.ci/scripts/test_espressif.sh:156-168).
    • test_implementation_conv2d asserts lowering before running the emulator.
    • test_implementation deletes a stale emulator.log and asserts each step's exit code.
    • _weighted_multipliers rejects reductions whose float32 reference could overflow.
    • Staged-batch, staged no-bias and stride-remainder cases were added.
  • Downgraded:
    • The quantizer still gives dilated and runtime-weight Conv2d an int32 bias that lowering then rejects (esp_fused_quantizer.py:55-60).
    • The only effect is that biases on those fallback convs are rounded to a multiple of s_in·s_w. Saturation would need |bias| > 2^19·s_in, given the 2^-12 observer eps. This no longer blocks.
  • No longer raised:
    • The non-mutable scratch schema. Planning keeps scratch disjoint from input and output, and test_dialect_conv2d asserts this.
    • A scratch target mismatch. It fails safely and logs the scratch.nbytes() check.
    • The unasserted linear and groups exports in test_convolution_falls_back_without_sizing. Their test_dialect op counts cover them.
  • Still open: see the sections below.

I couldn't run the tests here because Python execution needed approval in this job. The ESP-NN sources aren't in the checkout either, so claims about ESP-NN internals are marked low confidence.

Code Quality

  • Export aborts instead of falling back when the two scratch bounds disagree (backends/espressif/passes/lower_to_esp.py:510-518).

    • size_query is called outside the try at 504-509.
    • Lowering's own scratch_bound (481-498) is looser than the binding's check_conv_arithmetic (bindings.cpp:78-84). The binding counts input and filter twice, uses max(O, C) filter channels and adds an output term.
    • A shape that passes lowering's bound but fails the binding's therefore raises the pybind ValueError out of LowerToEsp.
    • Trigger: Conv2d(32768, 1, 1) on a (1, 32768, 1, 1) input. Lowering's bound is 1,491,223; the binding's is 2,150,613,248, above INT32_MAX.
    • Lines 517-518 also raise RuntimeError for any vendor size above lowering's bound, even with allow_cpu_fallback=True. I couldn't check whether any ESP-NN path returns a size that large (low confidence).
    • Both paths contradict README:45-46 ("Other convolution configurations retain portable execution").
    • The shapes are extreme, so this is minor, but the fix is small:
      • query inside the try and return None on ValueError;
      • keep one bound instead of three diverging copies (bindings.cpp:50-85, lower_to_esp.py:481-498, op_quantized_conv2d.cpp:79-112).

    Fix this →

  • The device kernel trusts the PTE's convolution geometry (backends/espressif/ops/op_quantized_conv2d.cpp:45-112).

    • output_h, output_w and outputs are read from out. The kernel never checks any of these:
      • out.size(0) == batches;
      • weight.size(0) == outputs;
      • weight.size(3) == channels;
      • output_h == (height + 2*padding[0] - kernel_h) / stride[0] + 1, or the width equivalent.
    • The nbytes checks keep the outer tensors in bounds. ESP-NN then iterates an output grid that need not match the input, stride and padding.
    • The int32 guard at 93-112 is also weaker than the host's (bindings.cpp:78-80). It bounds the padded height by height + padding[0] rather than + 2*padding[0], and it has no output term.
    • So a malformed PTE can reach vendor int32 arithmetic that the host guard was written to exclude.
    • Lowering already enforces the geometry (lower_to_esp.py:466-480), so this only matters for malformed or hand-edited PTEs. Mirroring that check here is defense in depth.

    Fix this →

  • Fallback reasons are dropped (lower_to_esp.py:560-570).

    • _lower_conv2d returns False silently when the binding or query is missing, and _conv2d_kernel_args returns a bare None for every rejection.
    • With the default recipe, users silently get portable convolution.
    • In strict mode, users only see ...left these operators on the CPU: forward.… (aten::convolution) (lowering.py:114-119), with no hint to build with EXECUTORCH_BUILD_ESP_NN_PYBINDS=ON. _lower_linear (542-549) already reports its reason.
    • The except AttributeError probe at 564-567 is unreachable with the in-tree binding, which always defines conv2d_scratch_size (bindings.cpp:185-192). It exists only for the monkeypatch.delattr test, and it is the dynamic-attribute pattern that CLAUDE.md discourages.

    Fix this →

API Design

  • The README overstates the scratch alignment contract (backends/espressif/README.md:63-64).

    • It says "ordinary memory planning assigns its storage and 16-byte alignment". MemoryPlanningPass only aligns offsets. The base address is whatever planned buffer the application passes, and ExecuTorch doesn't require it to be aligned.
    • The in-tree runners do align: et_model.cpp:233-238 checks it, and esp_executor_runner.cpp:188-205 uses aligned(16). The README never tells other applications to do the same.
    • The kernel passes the pointer to esp_nn_set_conv_scratch_buf unchecked (op_quantized_conv2d.cpp:137).
    • If ESP-NN's S3 SIMD paths need aligned scratch, an unaligned buffer gives silently wrong outputs. This is low confidence because I couldn't check the vendor code, though the delegate's ARENA_ALIGNMENT = 16 (int8_interpreter.cpp:31) suggests they do.
    • Fix: state the requirement in the README, and ET_KERNEL_CHECK that a non-empty scratch pointer is 16-byte aligned.

    Fix this →

Thread Safety

  • The operator library sets and clears ESP-NN's global conv scratch pointer without the delegate's lock (op_quantized_conv2d.cpp:137-150).

    • The delegate serializes on invoke_mutex()/InvokeLock (int8_interpreter.cpp:34-53, 1677-1679). Both live in an anonymous namespace, so the operator library can't take them.
    • README:66-68 documents the hazard. Even so, every application that mixes the delegate with operator-library models, or runs two operator-library models in different tasks, has to wrap each inference in its own mutex.
    • If it doesn't, one esp_nn_conv_s8 can run against another call's scratch, sized for a different shape, or against nullptr after line 150 resets the pointer. The result is out-of-bounds writes or a crash.
    • Sharing the existing lock through a header that both sides include, and holding it across 137-150, makes the documented contract hold by construction.

    Fix this →

Testing

  • Common convolution shapes have no emulator coverage (backends/espressif/test/ops/test_conv2d.py:110-180).

    • Every 1x1 case uses stride 1, padding 0 and a bias. There is no 1x1 stride-2 case (ResNet/MobileNet downsample) and no 1x1 bias=False case, so the pointwise path's stride and null-bias handling is untested.
    • Non-1x1 kernels only use C ∈ {1, 3, 4, 5, 16}. Channel tiling beyond one 16-channel block is only exercised for 1x1 (C=256), and a partial trailing block such as C=24 is never exercised.
    • No case has padding ≥ 2, for example 5x5 with pad 2.
    • Each of these is a one-line test_cases entry.

    Fix this →

  • EspTester.test_implementation forwards target but still links the S3-only runtime (backends/espressif/test/tester.py:210-230).

    • It passes -DIDF_TARGET/--chip from self.target.
    • ET_BUILD_DIR_PATH, however, points at the runtime that CI builds only with toolchain-esp32s3.cmake (test_espressif.sh:74-75).
    • So a non-S3 tester links S3 objects into its firmware, and the P4/S31 plans in test_public_recipe_conv2d (263-264) are only ever checked on the host.
    • Fix: fail clearly for non-S3 targets, or build the runtime per target.

    Fix this →

Recommendation

Approve

Nothing is blocking. Two small fixes are worth folding in before merge:

  • catch the binding's ValueError in _conv2d_kernel_args;
  • correct the README alignment wording.

The kernel geometry checks, the shared ESP-NN lock and the extra coverage can follow later in stack #25158.

Review steps
  • Load the /pr-review skill and checklist
  • Diff 5ae7977 against origin/codex/espressif-scratch-bindings (1 commit, 14 files, +978/−80)
  • Run three independent reviewers
  • Consolidate findings
  • Re-check previous findings against the updated code
  • Fact-check and post

@rascani
rascani force-pushed the codex/espressif-fused-quant-conv2d branch from 5ae7977 to c8a3bf3 Compare November 17, 2026 22:05
@rascani
rascani force-pushed the codex/espressif-fused-quant-conv2d branch from c8a3bf3 to 472cb3b Compare November 17, 2026 23:40
@rascani
rascani force-pushed the codex/espressif-fused-quant-conv2d branch from 472cb3b to 6adbe1f Compare November 18, 2026 00:39
@rascani

rascani commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

Please re-review the current revision (4d6c92bba1afc81feb79b92a656045f94ab6ddf1) against this PR's base branch, codex/espressif-scratch-bindings. This PR is part of stack #25158. Re-check previous findings against the updated code and focus on actionable correctness, quantization/numerical, memory-safety, and regression-test issues in this PR.

Scratch size-query ValueError now retains the portable FQ convolution before weighted constants are inserted. A regression using an actual rejected vendor dimension bound checks numerical fallback, serialization, and strict-export rejection. Please review the ordering, exception scope, and coverage. Focused convolution tests passed, and 969 combined host tests passed at the stack tip with all four sizing bindings rebuilt. A separate shared-FQ fallback issue for a single output channel producing nonconstant per-tensor qparams remains known and unchanged by this fix.

@claude

claude Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Claude finished @rascani's task in 25m 3s —— View job


Re-review of 4d6c92b: the reordered size query and its except ValueError scope are correct and nothing blocks merge, but the new regression never checks that lowering reached the query, and nothing tests the RuntimeError that now sits inside the same try.

Detailed review

PR Review: #25154

Summary

This PR lowers fused_quant.convolution to esp_nn::quantized_conv2d_int8 with an export-planned scratch tensor. 4d6c92b moves the scratch-size query ahead of _weighted_constants and inside the try, so a binding ValueError now keeps portable convolution. I found no issues with the new ordering or with what the except ValueError catches. The remaining problems are minor.

Previous findings (review of 5ae7977), re-checked:

  • Fixed: a binding ValueError no longer aborts export (lower_to_esp.py:506-520). A Conv2d(32768, 2, 1) regression covers it.
  • Partly fixed: the README no longer says memory planning aligns the scratch, but it still doesn't state the 16-byte requirement (API Design).
  • Still open:
    • the shared ESP-NN lock (Thread Safety);
    • the kernel's geometry checks and the three diverging bound formulas, and the silent fallback with its AttributeError probe (Code Quality);
    • the common-shape and non-S3 emulator gaps (Testing).
  • Not re-raised: the int32 bias the quantizer gives to convs that never lower.

I couldn't run the tests because Python execution needs approval in this job. The ESP-NN 1.5.0 sources aren't in the checkout, so claims about vendor internals are marked low confidence.

Code Quality

  • Host planning and the device check call the vendor scratch getter with different arguments (ops/op_quantized_conv2d.cpp:114-131 vs scratch/bindings.cpp:99-105).

    • The binding passes extra = 0 and a conv_params_t with zero offsets and activation {0, 0}.
    • The kernel passes extra = 1, the real offsets and {-128, 127}, as the delegate does (espnn_runtime/src/int8_interpreter.cpp:504-506,571-579).
    • If any profile's getter reads these fields, the plan can be smaller than the device requires, and every call then fails the capacity check at 132-136. It fails closed, but P4/S31 (riscv_pie) only have host coverage. Low confidence: I couldn't read the vendor getters.
    • Fix: build the arguments the same way on both sides, for example extra = 1 and {-128, 127} in conv2d_scratch_size.

    Fix this →

  • Still open: the kernel trusts the PTE's convolution geometry (ops/op_quantized_conv2d.cpp:45-112).

    • It never checks weight.size(0) == outputs, weight.size(3) == channels, out.size(0) == batches, or output_h/output_w == (in + 2·pad − k) / stride + 1. Portable conv checks these with check_convolution_args and get_convolution_out_target_size (kernels/portable/cpu/util/kernel_ops_util.h:515,527).
    • Its int32 guard (93-112) is a third formula, alongside lowering's scratch_bound and the binding's check_conv_arithmetic. The new regression's shape (C=32768, 1x1, 2 outputs) passes the device guard at about 1.43 MB, although the host binding rejects it as overflow-prone.
    • Lowering enforces the geometry (lower_to_esp.py:468-482), so only a malformed PTE reaches the vendor code this way.

    Fix this →

  • Still open: conv fallback reasons are dropped (passes/lower_to_esp.py:519-520,563-569).

    • The new except ValueError: return None also discards the binding's message.
    • A missing binding or query falls back without a warning, and strict export only names aten::convolution.
    • The except AttributeError probe at 566-569 is only reachable through the monkeypatch.delattr test, because the binding always defines conv2d_scratch_size.

    Fix this →

Testing

  • The new regression doesn't pin what 4d6c92b changed (test/ops/test_conv2d.py:334-353).

    • It never shows that lowering reached the query. The test checks that the binding rejects these dims, and separately that the model falls back. Today the shape passes every earlier check in _conv2d_kernel_args. If a later change rejected it earlier (for example a tighter scratch_bound), the test would stay green while the except ValueError path lost its coverage.
      • Fix: after the direct call, wrap the real query with monkeypatch.setattr(s3_queries, "conv2d_scratch_size", spy), and assert the spy was called and raised. test_convolution_falls_back_without_sizing[query] already patches this module.
    • Nothing tests that the RuntimeError at lower_to_esp.py:514-515 escapes the try. It now sits inside the same try, so widening the except would silently turn it into a fallback. It aborts export even with allow_cpu_fallback=True, and since the query now runs first, it would also fire for convs that _weighted_constants would otherwise send to fallback (for example one with a runtime bias).
      • Fix: add a case whose patched query returns -1, and assert the RuntimeError.

    Fix this →

  • Still open: emulator coverage gaps (test/ops/test_conv2d.py:110-180, test/tester.py:215).

    • No case covers a 1x1 conv with stride 2 or bias=False, a non-1x1 kernel with more than 16 input channels, or padding ≥ 2.
    • test_implementation still links the S3-only runtime build for every target.

    Fix this →

API Design

  • The 16-byte scratch alignment requirement is neither documented nor checked (README.md:52, ops/op_quantized_conv2d.cpp:137).

    • The PR description says the application supplies an aligned arena. The README says only "Scratch storage uses the planned arena", and memory planning aligns offsets, not the buffer base.
    • The kernel hands the pointer to esp_nn_set_conv_scratch_buf unchecked.
    • Low confidence that ESP-NN needs the alignment. The delegate allocates with ARENA_ALIGNMENT = 16 (int8_interpreter.cpp:31), but the binding's 256 bytes of slack may already cover internal realignment.

    Fix this →

Thread Safety

  • Still open: the kernel sets and clears ESP-NN's global conv scratch pointer without the delegate's lock (ops/op_quantized_conv2d.cpp:137,150).

    • The delegate holds InvokeLock for a whole invoke, including the vTaskDelay(1) yield (int8_interpreter.cpp:1677-1702). The lock lives in an anonymous namespace (:29-53), so the operator library can't take it.
    • README:52-53 leaves this to the application. If an application doesn't serialize, an operator-library conv on another task can repoint or null the pointer in the middle of a delegate invoke.

    Fix this →

Recommendation

Approve

Nothing blocks merge. Worth folding in here:

  • the spy and RuntimeError cases for the new regression;
  • matching getter arguments on host and device.

The remaining open items can follow in stack #25158.

Review steps
  • Load the /pr-review skill and checklist
  • Diff 4d6c92b against origin/codex/espressif-scratch-bindings (2 commits, 14 files, +991/−80)
  • Run three independent reviewers
  • Consolidate findings
  • Re-check previous findings against the updated code
  • Fact-check and post

@JakeStevens

Copy link
Copy Markdown
Contributor

actual ci errors here

Lower static groups=1, unit-dilation int8 convolution using the ESP-NN
scratch-size bindings from the preceding commit. EspTarget and
EspKernelMode select the query used to allocate a planned byte tensor;
the runtime checks its size before setting ESP-NN's scratch pointer.

Support constant per-channel weights, derived int32 bias, batches,
rectangular kernels, stride, and padding. Unsupported calls or missing
sizing bindings keep portable convolution. Keep the delegate and shared
Fused Quant transforms unchanged.

Validated 208 backend Python tests, all 24 Conv2d ESP32-S3 emulator tests,
and lintrunner. Emulator cases cover repeated inference, undersized
scratch rejection, ANSI zero scratch, and missing-binding fallback.

Authored with OpenAI Codex.
Treat a sizing binding's bounds rejection as an unsupported convolution configuration. Query before inserting requantization constants so rejected calls remain available to portable decomposition without partial lowering state.

Added a real sizing-bound regression that checks numerical fallback, serialization, and strict-export rejection. Validated 34 host convolution tests with freshly built sizing bindings.

Authored with OpenAI Codex.
Keep checks that remove the S3 sizing binding or exercise its bounds explicitly on S3. Make the device fallback case remove sizing for whichever chip the CI job selects, preserving the target support introduced on main.

Authored with OpenAI Codex.
Exercise selected-target scratch planning and serialization with compute coverage required. Supported convolutions retain their reported layout copies; missing sizing or unsupported bounds still fail strict export.

Authored with OpenAI Codex.

This branch was previously deployed

1 inactive deployment
cadence — f0ccd798 Deployed Nov 21, 2026 by rascani via hifi-op-test / hifi4 #39853
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants