Skip to content

Commit 967de28

Browse files
committed
[Espressif] Lower Fused Quant Conv2d with planned scratch
Lower static groups=1, unit-dilation int8 convolution using the ESP-NN scratch-size bindings from the preceding commit. EspTarget and EspKernelMode select the query used to allocate a planned byte tensor; the runtime checks its size before setting ESP-NN's scratch pointer. Support constant per-channel weights, derived int32 bias, batches, rectangular kernels, stride, and padding. Unsupported calls or missing sizing bindings keep portable convolution. Keep the delegate and shared Fused Quant transforms unchanged. Validated 208 backend Python tests, all 24 Conv2d ESP32-S3 emulator tests, and lintrunner. Emulator cases cover repeated inference, undersized scratch rejection, ANSI zero scratch, and missing-binding fallback. Authored with OpenAI Codex.
1 parent 3bd9062 commit 967de28

14 files changed

Lines changed: 931 additions & 78 deletions

‎.ci/scripts/test_espressif.sh‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ python examples/espressif/export_espnn_test_programs.py \
6868
espnn_ops=$(tr -d '\n' < "${build_dir}/espnn-tests/operators.txt" \
6969
| tr ',' '\n' | grep -v '^quantized_decomposed::' | paste -sd, -)
7070
if [[ "${espnn_kernels}" == optimized ]]; then
71-
espnn_ops="${espnn_ops},aten::sin.out,aten::relu.out,aten::max_pool2d_with_indices.out,aten::avg_pool2d.out,aten::permute_copy.out"
71+
espnn_ops="${espnn_ops},aten::sin.out,aten::relu.out,aten::max_pool2d_with_indices.out,aten::avg_pool2d.out,aten::permute_copy.out,aten::convolution.out"
7272
fi
7373

7474
cmake --preset esp-baremetal -G Ninja -B "${build_dir}/runtime" \
@@ -166,8 +166,12 @@ if [[ "${espnn_kernels}" == optimized ]]; then
166166
-x -v --tb=short backends/espressif/test/ops -k implementation \
167167
2>&1 | tee "${build_dir}/operator-library-tests.log"
168168
for sdkconfig in "${build_dir}"/operator-library-*/sdkconfig; do
169-
if ! grep -qx 'CONFIG_NN_OPTIMIZED=y' "${sdkconfig}"; then
170-
echo "Expected CONFIG_NN_OPTIMIZED=y in ${sdkconfig}" >&2
169+
expected_config=CONFIG_NN_OPTIMIZED
170+
if [[ "${sdkconfig}" == *-ansi/sdkconfig ]]; then
171+
expected_config=CONFIG_NN_ANSI_C
172+
fi
173+
if ! grep -qx "${expected_config}=y" "${sdkconfig}"; then
174+
echo "Expected ${expected_config}=y in ${sdkconfig}" >&2
171175
exit 1
172176
fi
173177
done

‎backends/espressif/CMakeLists.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ idf_component_register(
1111
"ops/op_quantized_max_pool2d.cpp"
1212
"ops/op_quantized_avg_pool2d.cpp"
1313
"ops/op_quantized_linear.cpp"
14+
"ops/op_quantized_conv2d.cpp"
1415
PRIV_REQUIRES
1516
esp-nn
1617
WHOLE_ARCHIVE

‎backends/espressif/README.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,33 @@ per-output-channel symmetric int8 weights, and optional constant int32 bias.
3939
Leading input dimensions are treated as batch rows by ESP-NN's batched fully
4040
connected API. Unsupported linear calls fail at export with the reason;
4141
portable linear fallback is not available yet, even when CPU fallback is enabled.
42+
Conv2d supports static 4-D int8 inputs, constant per-output-channel symmetric
43+
weights, optional derived int32 bias, `groups=1`, and unit dilation. Batches,
44+
rectangular kernels, stride and padding are supported. Other convolution
45+
configurations retain portable execution.
46+
47+
Set the recipe's `EspTarget` and `EspKernelMode` to match the device's ESP-NN
48+
configuration when planning scratch:
49+
50+
```python
51+
recipe = get_esp_fused_quant_recipe(
52+
target=EspTarget.ESP32P4, kernel_mode=EspKernelMode.OPTIMIZED
53+
)
54+
```
55+
56+
The defaults are `EspTarget.ESP32S3` and `EspKernelMode.OPTIMIZED`.
57+
Accelerated Conv2d export requires the optional [scratch-size bindings](#scratch-size-queries).
58+
If the selected binding or query is unavailable, convolution retains portable
59+
execution; strict export rejects that fallback. Other operator exports do not
60+
require the bindings.
61+
62+
Lowering adds a scratch Tensor to each convolution call, and ordinary memory
63+
planning assigns its storage and 16-byte alignment. The device checks its
64+
required size before using the buffer. Scratch follows the planned arena's
65+
memory placement. ESP-NN uses a global scratch pointer: applications must
66+
serialize operator-library inference and must not run it concurrently with
67+
the delegate or other ESP-NN calls.
68+
4269
The shared Fused Quant layout pass introduces contiguous NHWC tensors for
4370
spatial operators; model inputs and outputs retain their original layout.
4471
Required layout copies are visible in the operator-library report and accepted

‎backends/espressif/lowering.py‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
from executorch.backends.espressif.partition.espnn_partitioner import EspNNPartitioner
1717
from executorch.backends.espressif.passes import EspPassManager
1818
from executorch.backends.espressif.quantizer import EspFusedQuantizer
19+
from executorch.backends.espressif.target import EspKernelMode, EspTarget
1920
from executorch.exir import (
2021
EdgeCompileConfig,
2122
EdgeProgramManager,
@@ -76,6 +77,7 @@ def get_operator_library_report(edge: EdgeProgramManager) -> dict:
7677
in {
7778
"esp_nn::quantized_max_pool2d_int8",
7879
"esp_nn::quantized_avg_pool2d_int8",
80+
"esp_nn::quantized_conv2d_int8",
7981
}
8082
for neighbor in neighbors
8183
):
@@ -120,13 +122,17 @@ def get_esp_fused_quant_recipe(
120122
*,
121123
quantizer: FusedQuantQuantizer | None = None,
122124
allow_cpu_fallback: bool = True,
125+
target: EspTarget = EspTarget.ESP32S3,
126+
kernel_mode: EspKernelMode = EspKernelMode.OPTIMIZED,
123127
) -> ExportRecipe:
124128
"""Quantize and lower a float model for the Espressif operator library.
125129
126130
Use with executorch.export.export(). Supported Fused Quant operators become
127131
direct ESP-NN calls; other operators decompose to portable kernels. Setting
128132
allow_cpu_fallback=False rejects remaining compute, excluding Q/DQ and required NHWC layout copies.
129133
Unsupported linear calls currently raise even when CPU fallback is enabled.
134+
Set target and kernel_mode to match the device's ESP-NN configuration
135+
when planning scratch.
130136
The EspNNBackend delegate continues to use EspNNQuantizer and to_espnn_edge.
131137
"""
132138
quantizer = quantizer if quantizer is not None else EspFusedQuantizer()
@@ -135,7 +141,7 @@ def fuse(graph_module):
135141
quantizer.fuse(graph_module)
136142
return graph_module
137143

138-
transforms = [lambda _: EspPassManager()]
144+
transforms = [lambda _: EspPassManager(target=target, kernel_mode=kernel_mode)]
139145
if not allow_cpu_fallback:
140146
transforms.append(_require_no_portable_compute)
141147
return ExportRecipe(
Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
/*
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
* All rights reserved.
4+
*
5+
* This source code is licensed under the BSD-style license found in the
6+
* LICENSE file in the root directory of this source tree.
7+
*/
8+
9+
#include <executorch/runtime/kernel/kernel_includes.h>
10+
11+
#include <sdkconfig.h>
12+
13+
#include <esp_nn.h>
14+
15+
#include <algorithm>
16+
#include <cstdint>
17+
#include <limits>
18+
19+
namespace espressif::native {
20+
21+
using executorch::aten::IntArrayRef;
22+
using executorch::aten::Tensor;
23+
using executorch::runtime::KernelRuntimeContext;
24+
25+
// cppcheck-suppress unusedFunction
26+
Tensor& quantized_conv2d_int8_out(
27+
KernelRuntimeContext& ctx,
28+
const Tensor& input,
29+
const Tensor& weight,
30+
const std::optional<Tensor>& bias,
31+
IntArrayRef stride,
32+
IntArrayRef padding,
33+
const Tensor& output_multiplier,
34+
const Tensor& output_shift,
35+
int64_t input_offset,
36+
int64_t output_offset,
37+
const Tensor& scratch,
38+
Tensor& out) {
39+
ET_KERNEL_CHECK(
40+
ctx,
41+
input.dim() == 4 && weight.dim() == 4 && out.dim() == 4 &&
42+
stride.size() == 2 && padding.size() == 2,
43+
InvalidArgument,
44+
out);
45+
const auto batches = input.size(0);
46+
const auto height = input.size(1);
47+
const auto width = input.size(2);
48+
const auto channels = input.size(3);
49+
const auto kernel_h = weight.size(1);
50+
const auto kernel_w = weight.size(2);
51+
const auto output_h = out.size(1);
52+
const auto output_w = out.size(2);
53+
const auto outputs = out.size(3);
54+
const int64_t dimensions[] = {
55+
height,
56+
width,
57+
channels,
58+
kernel_h,
59+
kernel_w,
60+
output_h,
61+
output_w,
62+
outputs,
63+
stride[0],
64+
stride[1]};
65+
for (const auto value : dimensions) {
66+
ET_KERNEL_CHECK(
67+
ctx,
68+
value > 0 && value <= std::numeric_limits<uint16_t>::max(),
69+
InvalidArgument,
70+
out);
71+
}
72+
for (const auto value : padding) {
73+
ET_KERNEL_CHECK(
74+
ctx,
75+
value >= 0 && value <= std::numeric_limits<uint16_t>::max(),
76+
InvalidArgument,
77+
out);
78+
}
79+
const uint64_t input_bytes = uint64_t(height) * width * channels;
80+
const uint64_t output_bytes = uint64_t(output_h) * output_w * outputs;
81+
const uint64_t filter_bytes =
82+
uint64_t(kernel_h) * kernel_w * channels * outputs;
83+
const uint64_t padded_h = std::max<int64_t>(
84+
height + padding[0], (output_h - 1) * stride[0] + kernel_h);
85+
const uint64_t padded_w = std::max<int64_t>(
86+
width + padding[1], (output_w - 1) * stride[1] + kernel_w);
87+
const uint64_t aligned_channels = (uint64_t(channels) + 15) & ~uint64_t(15);
88+
const uint64_t aligned_filter_bytes =
89+
(uint64_t(kernel_w) * aligned_channels + 15) * kernel_h * outputs;
90+
const uint64_t staging_bytes =
91+
40 * ((uint64_t(kernel_h) * kernel_w * channels + 15) & ~uint64_t(15));
92+
constexpr auto limit = std::numeric_limits<int32_t>::max();
93+
ET_KERNEL_CHECK(
94+
ctx,
95+
batches >= 0 && input_bytes <= limit && output_bytes <= limit &&
96+
filter_bytes <= weight.nbytes() && aligned_filter_bytes <= limit &&
97+
staging_bytes <= limit &&
98+
padded_h <= limit / (padded_w * aligned_channels) &&
99+
uint64_t(batches) <= input.nbytes() / input_bytes &&
100+
uint64_t(batches) <= out.nbytes() / output_bytes &&
101+
uint64_t(outputs) <= output_multiplier.nbytes() / sizeof(int32_t) &&
102+
uint64_t(outputs) <= output_shift.nbytes() / sizeof(int32_t) &&
103+
(!bias || uint64_t(outputs) <= bias->nbytes() / sizeof(int32_t)),
104+
InvalidArgument,
105+
out);
106+
ET_KERNEL_CHECK(
107+
ctx,
108+
padded_h * padded_w * aligned_channels + aligned_filter_bytes +
109+
staging_bytes + 8 * outputs + 16 * 1024 + 256 <=
110+
limit,
111+
InvalidArgument,
112+
out);
113+
114+
const data_dims_t input_dims{
115+
int32_t(width), int32_t(height), int32_t(channels), 1};
116+
const data_dims_t filter_dims{
117+
int32_t(kernel_w), int32_t(kernel_h), int32_t(channels), 1};
118+
const data_dims_t output_dims{
119+
int32_t(output_w), int32_t(output_h), int32_t(outputs), 1};
120+
const conv_params_t params{
121+
int32_t(input_offset),
122+
int32_t(output_offset),
123+
{int32_t(stride[1]), int32_t(stride[0])},
124+
{int32_t(padding[1]), int32_t(padding[0])},
125+
{1, 1},
126+
{-128, 127}};
127+
const quant_data_t quant{
128+
const_cast<int32_t*>(output_shift.const_data_ptr<int32_t>()),
129+
const_cast<int32_t*>(output_multiplier.const_data_ptr<int32_t>())};
130+
const auto required = esp_nn_get_conv_scratch_size(
131+
&input_dims, &filter_dims, &output_dims, &params);
132+
ET_KERNEL_CHECK(
133+
ctx,
134+
required >= 0 && size_t(required) <= scratch.nbytes(),
135+
InvalidArgument,
136+
out);
137+
esp_nn_set_conv_scratch_buf(scratch.mutable_data_ptr<uint8_t>());
138+
for (int64_t batch = 0; batch < batches; ++batch) {
139+
esp_nn_conv_s8(
140+
&input_dims,
141+
input.const_data_ptr<int8_t>() + batch * input_bytes,
142+
&filter_dims,
143+
weight.const_data_ptr<int8_t>(),
144+
bias ? bias->const_data_ptr<int32_t>() : nullptr,
145+
&output_dims,
146+
out.mutable_data_ptr<int8_t>() + batch * output_bytes,
147+
&params,
148+
&quant);
149+
}
150+
esp_nn_set_conv_scratch_buf(nullptr);
151+
return out;
152+
}
153+
154+
} // namespace espressif::native

‎backends/espressif/ops/operators.py‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,17 @@
5858
"*, Tensor(a!) out) -> Tensor(a!)"
5959
)
6060

61+
lib.define(
62+
"quantized_conv2d_int8(Tensor input, Tensor weight, Tensor? bias, "
63+
"int[2] stride, int[2] padding, Tensor output_multiplier, Tensor output_shift, "
64+
"int input_offset, int output_offset, Tensor scratch) -> Tensor"
65+
)
66+
lib.define(
67+
"quantized_conv2d_int8.out(Tensor input, Tensor weight, Tensor? bias, "
68+
"int[2] stride, int[2] padding, Tensor output_multiplier, Tensor output_shift, "
69+
"int input_offset, int output_offset, Tensor scratch, *, Tensor(a!) out) -> Tensor(a!)"
70+
)
71+
6172

6273
@register_fake("esp_nn::quantized_max_pool2d_int8")
6374
@register_fake("esp_nn::quantized_avg_pool2d_int8")
@@ -194,3 +205,60 @@ def quantized_linear_int8(
194205
dim=-1,
195206
)
196207
return (result + output_offset).clamp(-128, 127).to(torch.int8)
208+
209+
210+
@register_fake("esp_nn::quantized_conv2d_int8")
211+
def quantized_conv2d_int8_meta(
212+
input,
213+
weight,
214+
bias,
215+
stride,
216+
padding,
217+
output_multiplier,
218+
output_shift,
219+
input_offset,
220+
output_offset,
221+
scratch,
222+
):
223+
batch, height, width, _ = input.shape
224+
outputs, kernel_h, kernel_w, _ = weight.shape
225+
return input.new_empty(
226+
(
227+
batch,
228+
(height + 2 * padding[0] - kernel_h) // stride[0] + 1,
229+
(width + 2 * padding[1] - kernel_w) // stride[1] + 1,
230+
outputs,
231+
)
232+
)
233+
234+
235+
@impl(lib, "quantized_conv2d_int8", "CompositeExplicitAutograd")
236+
def quantized_conv2d_int8(
237+
input,
238+
weight,
239+
bias,
240+
stride,
241+
padding,
242+
output_multiplier,
243+
output_shift,
244+
input_offset,
245+
output_offset,
246+
scratch,
247+
):
248+
centered = input.to(torch.int64).permute(0, 3, 1, 2) + input_offset
249+
padded = torch.nn.functional.pad(
250+
centered, (padding[1], padding[1], padding[0], padding[0])
251+
)
252+
windows = padded.unfold(2, weight.shape[1], stride[0]).unfold(
253+
3, weight.shape[2], stride[1]
254+
)
255+
windows = windows.permute(0, 2, 3, 4, 5, 1).flatten(3)
256+
return quantized_linear_int8(
257+
windows,
258+
weight.flatten(1),
259+
bias,
260+
output_multiplier,
261+
output_shift,
262+
0,
263+
output_offset,
264+
)

‎backends/espressif/ops/operators.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,3 +33,9 @@
3333
kernels:
3434
- arg_meta: null
3535
kernel_name: espressif::quantized_linear_int8_out
36+
37+
- func: esp_nn::quantized_conv2d_int8.out(Tensor input, Tensor weight, Tensor? bias, int[2] stride, int[2] padding, Tensor output_multiplier, Tensor output_shift, int input_offset, int output_offset, Tensor scratch, *, Tensor(a!) out) -> Tensor(a!)
38+
variants: function
39+
kernels:
40+
- arg_meta: null
41+
kernel_name: espressif::quantized_conv2d_int8_out

‎backends/espressif/passes/esp_pass_manager.py‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66

77
import copy
88

9+
from executorch.backends.espressif.target import EspKernelMode, EspTarget
910
from executorch.backends.transforms.fused_quant.decompose_fused_quant import (
1011
AssertNoFusedQuantOps,
1112
DecomposeFusedQuant,
@@ -30,12 +31,17 @@
3031
class EspPassManager(ExportedProgramPassManager):
3132
"""Lower fused-quant Edge operators to ESP-NN or portable kernels."""
3233

33-
def __init__(self) -> None:
34+
def __init__(
35+
self,
36+
*,
37+
target: EspTarget = EspTarget.ESP32S3,
38+
kernel_mode: EspKernelMode = EspKernelMode.OPTIMIZED,
39+
) -> None:
3440
super().__init__(
3541
passes=[
3642
ToChannelsLast(),
3743
ExportPass(),
38-
FusedQuantToEspPass(),
44+
FusedQuantToEspPass(target=target, kernel_mode=kernel_mode),
3945
ToChannelsFirst(),
4046
DecomposeFusedQuant(),
4147
RemoveUnusedConstantsPass(),

0 commit comments

Comments
 (0)