Skip to content

Commit c8a3bf3

Browse files
committed
[Espressif] Lower Fused Quant Conv2d with planned scratch
Lower static groups=1, unit-dilation int8 convolution using the ESP-NN scratch-size bindings from the preceding commit. EspTarget and EspKernelMode select the query used to allocate a planned byte tensor; the runtime checks its size before setting ESP-NN's scratch pointer. Support constant per-channel weights, derived int32 bias, batches, rectangular kernels, stride, and padding. Unsupported calls or missing sizing bindings keep portable convolution. Keep the delegate and shared Fused Quant transforms unchanged. Validated 208 backend Python tests, all 24 Conv2d ESP32-S3 emulator tests, and lintrunner. Emulator cases cover repeated inference, undersized scratch rejection, ANSI zero scratch, and missing-binding fallback. Authored with OpenAI Codex.
1 parent 3b04a69 commit c8a3bf3

14 files changed

Lines changed: 978 additions & 80 deletions

‎.ci/scripts/test_espressif.sh‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ python examples/espressif/export_espnn_test_programs.py \
6868
espnn_ops=$(tr -d '\n' < "${build_dir}/espnn-tests/operators.txt" \
6969
| tr ',' '\n' | grep -v '^quantized_decomposed::' | paste -sd, -)
7070
if [[ "${espnn_kernels}" == optimized ]]; then
71-
espnn_ops="${espnn_ops},aten::sin.out,aten::relu.out,aten::max_pool2d_with_indices.out,aten::avg_pool2d.out,aten::permute_copy.out"
71+
espnn_ops="${espnn_ops},aten::sin.out,aten::relu.out,aten::max_pool2d_with_indices.out,aten::avg_pool2d.out,aten::permute_copy.out,aten::convolution.out"
7272
fi
7373

7474
cmake --preset esp-baremetal -G Ninja -B "${build_dir}/runtime" \
@@ -162,13 +162,19 @@ if [[ "${espnn_kernels}" == optimized ]]; then
162162
logged "scratch-build.log" cmake --build "${build_dir}/scratch-sizing"
163163
logged "scratch-tests.log" env EXECUTORCH_REQUIRE_ESP_NN_SCRATCH=1 \
164164
python -m pytest -q backends/espressif/test/test_scratch.py
165+
logged "operator-library-host-tests.log" python -m pytest -q \
166+
backends/espressif/test/ops -k 'not implementation'
165167
# Stop on failure so later cases cannot overwrite its firmware and logs.
166168
ESPRESSIF_BUILD_DIR="${build_dir}" python -m pytest \
167169
-x -v --tb=short backends/espressif/test/ops -k implementation \
168170
2>&1 | tee "${build_dir}/operator-library-tests.log"
169171
for sdkconfig in "${build_dir}"/operator-library-*/sdkconfig; do
170-
if ! grep -qx 'CONFIG_NN_OPTIMIZED=y' "${sdkconfig}"; then
171-
echo "Expected CONFIG_NN_OPTIMIZED=y in ${sdkconfig}" >&2
172+
expected_config=CONFIG_NN_OPTIMIZED
173+
if [[ "${sdkconfig}" == *-ansi/sdkconfig ]]; then
174+
expected_config=CONFIG_NN_ANSI_C
175+
fi
176+
if ! grep -qx "${expected_config}=y" "${sdkconfig}"; then
177+
echo "Expected ${expected_config}=y in ${sdkconfig}" >&2
172178
exit 1
173179
fi
174180
done

‎backends/espressif/CMakeLists.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ idf_component_register(
1111
"ops/op_quantized_max_pool2d.cpp"
1212
"ops/op_quantized_avg_pool2d.cpp"
1313
"ops/op_quantized_linear.cpp"
14+
"ops/op_quantized_conv2d.cpp"
1415
PRIV_REQUIRES
1516
esp-nn
1617
WHOLE_ARCHIVE

‎backends/espressif/README.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,33 @@ per-output-channel symmetric int8 weights, and optional constant int32 bias.
4040
Leading input dimensions are treated as batch rows by ESP-NN's batched fully
4141
connected API. Unsupported linear calls fail at export with the reason;
4242
portable linear fallback is not available yet, even when CPU fallback is enabled.
43+
Conv2d supports static 4-D int8 inputs, constant per-output-channel symmetric
44+
weights, optional derived int32 bias, `groups=1`, and unit dilation. Batches,
45+
rectangular kernels, stride and padding are supported. Other convolution
46+
configurations retain portable execution.
47+
48+
Set the recipe's `EspTarget` and `EspKernelMode` to match the device's ESP-NN
49+
configuration when planning scratch:
50+
51+
```python
52+
recipe = get_esp_fused_quant_recipe(
53+
target=EspTarget.ESP32P4, kernel_mode=EspKernelMode.OPTIMIZED
54+
)
55+
```
56+
57+
The defaults are `EspTarget.ESP32S3` and `EspKernelMode.OPTIMIZED`.
58+
Accelerated Conv2d export requires the optional [scratch-size bindings](#scratch-size-queries).
59+
If the selected binding or query is unavailable, convolution retains portable
60+
execution; strict export rejects that fallback. Other operator exports do not
61+
require the bindings.
62+
63+
Lowering adds a scratch Tensor to each convolution call, and ordinary memory
64+
planning assigns its storage and 16-byte alignment. The device checks its
65+
required size before using the buffer. Scratch follows the planned arena's
66+
memory placement. ESP-NN uses a global scratch pointer: applications must
67+
serialize operator-library inference and must not run it concurrently with
68+
the delegate or other ESP-NN calls.
69+
4370
The shared Fused Quant layout pass introduces contiguous NHWC tensors for
4471
spatial operators; model inputs and outputs retain their original layout.
4572
Required layout copies are visible in the operator-library report and accepted

‎backends/espressif/lowering.py‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
from executorch.backends.espressif.partition.espnn_partitioner import EspNNPartitioner
1717
from executorch.backends.espressif.passes import EspPassManager
1818
from executorch.backends.espressif.quantizer import get_esp_fused_quant_quantizer
19+
from executorch.backends.espressif.target import EspKernelMode, EspTarget
1920
from executorch.exir import (
2021
EdgeCompileConfig,
2122
EdgeProgramManager,
@@ -80,6 +81,7 @@ def get_operator_library_report(edge: EdgeProgramManager) -> dict:
8081
in {
8182
"esp_nn::quantized_max_pool2d_int8",
8283
"esp_nn::quantized_avg_pool2d_int8",
84+
"esp_nn::quantized_conv2d_int8",
8385
}
8486
for neighbor in neighbors
8587
):
@@ -124,13 +126,17 @@ def get_esp_fused_quant_recipe(
124126
*,
125127
quantizer: FusedQuantQuantizer | None = None,
126128
allow_cpu_fallback: bool = True,
129+
target: EspTarget = EspTarget.ESP32S3,
130+
kernel_mode: EspKernelMode = EspKernelMode.OPTIMIZED,
127131
) -> ExportRecipe:
128132
"""Quantize and lower a float model for the Espressif operator library.
129133
130134
Use with executorch.export.export(). Supported Fused Quant operators become
131135
direct ESP-NN calls; other operators decompose to portable kernels. Setting
132136
allow_cpu_fallback=False rejects remaining compute, excluding Q/DQ and required NHWC layout copies.
133137
Unsupported linear calls currently raise even when CPU fallback is enabled.
138+
Set target and kernel_mode to match the device's ESP-NN configuration
139+
when planning scratch.
134140
Use EspNNQuantizer and to_espnn_edge for the EspNNBackend delegate.
135141
"""
136142
quantizer = quantizer if quantizer is not None else get_esp_fused_quant_quantizer()
@@ -139,7 +145,7 @@ def fuse(graph_module):
139145
quantizer.fuse(graph_module)
140146
return graph_module
141147

142-
transforms = [lambda _: EspPassManager()]
148+
transforms = [lambda _: EspPassManager(target=target, kernel_mode=kernel_mode)]
143149
if not allow_cpu_fallback:
144150
transforms.append(_require_no_portable_compute)
145151
return ExportRecipe(
Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
/*
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
* All rights reserved.
4+
*
5+
* This source code is licensed under the BSD-style license found in the
6+
* LICENSE file in the root directory of this source tree.
7+
*/
8+
9+
#include <executorch/runtime/kernel/kernel_includes.h>
10+
11+
#include <sdkconfig.h>
12+
13+
#include <esp_nn.h>
14+
15+
#include <algorithm>
16+
#include <cstdint>
17+
#include <limits>
18+
19+
namespace espressif::native {
20+
21+
using executorch::aten::IntArrayRef;
22+
using executorch::aten::Tensor;
23+
using executorch::runtime::KernelRuntimeContext;
24+
25+
// cppcheck-suppress unusedFunction
26+
Tensor& quantized_conv2d_int8_out(
27+
KernelRuntimeContext& ctx,
28+
const Tensor& input,
29+
const Tensor& weight,
30+
const std::optional<Tensor>& bias,
31+
IntArrayRef stride,
32+
IntArrayRef padding,
33+
const Tensor& output_multiplier,
34+
const Tensor& output_shift,
35+
int64_t input_offset,
36+
int64_t output_offset,
37+
const Tensor& scratch,
38+
Tensor& out) {
39+
ET_KERNEL_CHECK(
40+
ctx,
41+
input.dim() == 4 && weight.dim() == 4 && out.dim() == 4 &&
42+
stride.size() == 2 && padding.size() == 2,
43+
InvalidArgument,
44+
out);
45+
const auto batches = input.size(0);
46+
const auto height = input.size(1);
47+
const auto width = input.size(2);
48+
const auto channels = input.size(3);
49+
const auto kernel_h = weight.size(1);
50+
const auto kernel_w = weight.size(2);
51+
const auto output_h = out.size(1);
52+
const auto output_w = out.size(2);
53+
const auto outputs = out.size(3);
54+
const int64_t dimensions[] = {
55+
height,
56+
width,
57+
channels,
58+
kernel_h,
59+
kernel_w,
60+
output_h,
61+
output_w,
62+
outputs,
63+
stride[0],
64+
stride[1]};
65+
for (const auto value : dimensions) {
66+
ET_KERNEL_CHECK(
67+
ctx,
68+
value > 0 && value <= std::numeric_limits<uint16_t>::max(),
69+
InvalidArgument,
70+
out);
71+
}
72+
for (const auto value : padding) {
73+
ET_KERNEL_CHECK(
74+
ctx,
75+
value >= 0 && value <= std::numeric_limits<uint16_t>::max(),
76+
InvalidArgument,
77+
out);
78+
}
79+
const uint64_t input_bytes = uint64_t(height) * width * channels;
80+
const uint64_t output_bytes = uint64_t(output_h) * output_w * outputs;
81+
const uint64_t filter_bytes =
82+
uint64_t(kernel_h) * kernel_w * channels * outputs;
83+
const uint64_t padded_h = std::max<int64_t>(
84+
height + padding[0], (output_h - 1) * stride[0] + kernel_h);
85+
const uint64_t padded_w = std::max<int64_t>(
86+
width + padding[1], (output_w - 1) * stride[1] + kernel_w);
87+
const uint64_t aligned_channels = (uint64_t(channels) + 15) & ~uint64_t(15);
88+
const uint64_t aligned_filter_bytes =
89+
(uint64_t(kernel_w) * aligned_channels + 15) * kernel_h * outputs;
90+
const uint64_t staging_bytes =
91+
40 * ((uint64_t(kernel_h) * kernel_w * channels + 15) & ~uint64_t(15));
92+
constexpr auto limit = std::numeric_limits<int32_t>::max();
93+
ET_KERNEL_CHECK(
94+
ctx,
95+
batches >= 0 && input_bytes <= limit && output_bytes <= limit &&
96+
filter_bytes <= weight.nbytes() && aligned_filter_bytes <= limit &&
97+
staging_bytes <= limit &&
98+
padded_h <= limit / (padded_w * aligned_channels) &&
99+
uint64_t(batches) <= input.nbytes() / input_bytes &&
100+
uint64_t(batches) <= out.nbytes() / output_bytes &&
101+
uint64_t(outputs) <= output_multiplier.nbytes() / sizeof(int32_t) &&
102+
uint64_t(outputs) <= output_shift.nbytes() / sizeof(int32_t) &&
103+
(!bias || uint64_t(outputs) <= bias->nbytes() / sizeof(int32_t)),
104+
InvalidArgument,
105+
out);
106+
ET_KERNEL_CHECK(
107+
ctx,
108+
padded_h * padded_w * aligned_channels + aligned_filter_bytes +
109+
staging_bytes + 8 * outputs + 16 * 1024 + 256 <=
110+
limit,
111+
InvalidArgument,
112+
out);
113+
114+
const data_dims_t input_dims{
115+
int32_t(width), int32_t(height), int32_t(channels), 1};
116+
const data_dims_t filter_dims{
117+
int32_t(kernel_w), int32_t(kernel_h), int32_t(channels), 1};
118+
const data_dims_t output_dims{
119+
int32_t(output_w), int32_t(output_h), int32_t(outputs), 1};
120+
const conv_params_t params{
121+
int32_t(input_offset),
122+
int32_t(output_offset),
123+
{int32_t(stride[1]), int32_t(stride[0])},
124+
{int32_t(padding[1]), int32_t(padding[0])},
125+
{1, 1},
126+
{-128, 127}};
127+
const quant_data_t quant{
128+
const_cast<int32_t*>(output_shift.const_data_ptr<int32_t>()),
129+
const_cast<int32_t*>(output_multiplier.const_data_ptr<int32_t>())};
130+
const auto required = esp_nn_get_conv_scratch_size(
131+
&input_dims, &filter_dims, &output_dims, &params);
132+
ET_KERNEL_CHECK(
133+
ctx,
134+
required >= 0 && size_t(required) <= scratch.nbytes(),
135+
InvalidArgument,
136+
out);
137+
esp_nn_set_conv_scratch_buf(scratch.mutable_data_ptr<uint8_t>());
138+
for (int64_t batch = 0; batch < batches; ++batch) {
139+
esp_nn_conv_s8(
140+
&input_dims,
141+
input.const_data_ptr<int8_t>() + batch * input_bytes,
142+
&filter_dims,
143+
weight.const_data_ptr<int8_t>(),
144+
bias ? bias->const_data_ptr<int32_t>() : nullptr,
145+
&output_dims,
146+
out.mutable_data_ptr<int8_t>() + batch * output_bytes,
147+
&params,
148+
&quant);
149+
}
150+
esp_nn_set_conv_scratch_buf(nullptr);
151+
return out;
152+
}
153+
154+
} // namespace espressif::native

‎backends/espressif/ops/operators.py‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,17 @@
5858
"*, Tensor(a!) out) -> Tensor(a!)"
5959
)
6060

61+
lib.define(
62+
"quantized_conv2d_int8(Tensor input, Tensor weight, Tensor? bias, "
63+
"int[2] stride, int[2] padding, Tensor output_multiplier, Tensor output_shift, "
64+
"int input_offset, int output_offset, Tensor scratch) -> Tensor"
65+
)
66+
lib.define(
67+
"quantized_conv2d_int8.out(Tensor input, Tensor weight, Tensor? bias, "
68+
"int[2] stride, int[2] padding, Tensor output_multiplier, Tensor output_shift, "
69+
"int input_offset, int output_offset, Tensor scratch, *, Tensor(a!) out) -> Tensor(a!)"
70+
)
71+
6172

6273
@register_fake("esp_nn::quantized_max_pool2d_int8")
6374
@register_fake("esp_nn::quantized_avg_pool2d_int8")
@@ -194,3 +205,60 @@ def quantized_linear_int8(
194205
dim=-1,
195206
)
196207
return (result + output_offset).clamp(-128, 127).to(torch.int8)
208+
209+
210+
@register_fake("esp_nn::quantized_conv2d_int8")
211+
def quantized_conv2d_int8_meta(
212+
input,
213+
weight,
214+
bias,
215+
stride,
216+
padding,
217+
output_multiplier,
218+
output_shift,
219+
input_offset,
220+
output_offset,
221+
scratch,
222+
):
223+
batch, height, width, _ = input.shape
224+
outputs, kernel_h, kernel_w, _ = weight.shape
225+
return input.new_empty(
226+
(
227+
batch,
228+
(height + 2 * padding[0] - kernel_h) // stride[0] + 1,
229+
(width + 2 * padding[1] - kernel_w) // stride[1] + 1,
230+
outputs,
231+
)
232+
)
233+
234+
235+
@impl(lib, "quantized_conv2d_int8", "CompositeExplicitAutograd")
236+
def quantized_conv2d_int8(
237+
input,
238+
weight,
239+
bias,
240+
stride,
241+
padding,
242+
output_multiplier,
243+
output_shift,
244+
input_offset,
245+
output_offset,
246+
scratch,
247+
):
248+
centered = input.to(torch.int64).permute(0, 3, 1, 2) + input_offset
249+
padded = torch.nn.functional.pad(
250+
centered, (padding[1], padding[1], padding[0], padding[0])
251+
)
252+
windows = padded.unfold(2, weight.shape[1], stride[0]).unfold(
253+
3, weight.shape[2], stride[1]
254+
)
255+
windows = windows.permute(0, 2, 3, 4, 5, 1).flatten(3)
256+
return quantized_linear_int8(
257+
windows,
258+
weight.flatten(1),
259+
bias,
260+
output_multiplier,
261+
output_shift,
262+
0,
263+
output_offset,
264+
)

‎backends/espressif/ops/operators.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,3 +33,9 @@
3333
kernels:
3434
- arg_meta: null
3535
kernel_name: espressif::quantized_linear_int8_out
36+
37+
- func: esp_nn::quantized_conv2d_int8.out(Tensor input, Tensor weight, Tensor? bias, int[2] stride, int[2] padding, Tensor output_multiplier, Tensor output_shift, int input_offset, int output_offset, Tensor scratch, *, Tensor(a!) out) -> Tensor(a!)
38+
variants: function
39+
kernels:
40+
- arg_meta: null
41+
kernel_name: espressif::quantized_conv2d_int8_out

‎backends/espressif/passes/esp_pass_manager.py‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66

77
import copy
88

9+
from executorch.backends.espressif.target import EspKernelMode, EspTarget
910
from executorch.backends.transforms.fused_quant.decompose_fused_quant import (
1011
AssertNoFusedQuantOps,
1112
DecomposeFusedQuant,
@@ -30,13 +31,18 @@
3031
class EspPassManager(ExportedProgramPassManager):
3132
"""Lower fused-quant Edge operators to ESP-NN or portable kernels."""
3233

33-
def __init__(self) -> None:
34+
def __init__(
35+
self,
36+
*,
37+
target: EspTarget = EspTarget.ESP32S3,
38+
kernel_mode: EspKernelMode = EspKernelMode.OPTIMIZED,
39+
) -> None:
3440
super().__init__(
3541
passes=[
3642
ToChannelsLast(),
3743
# Refresh contiguous tensor metadata after the layout rewrites.
3844
ExportPass(),
39-
LowerToEsp(),
45+
LowerToEsp(target=target, kernel_mode=kernel_mode),
4046
ToChannelsFirst(),
4147
DecomposeFusedQuant(),
4248
RemoveUnusedConstantsPass(),

0 commit comments

Comments
 (0)