Skip to content

Commit 9b4b2bc

Browse files
committed
[Espressif] Lower fused-quant AvgPool2d to ESP-NN
Share AvgPool2d input/output observers in get_esp_fused_quant_quantizer and lower supported static int8 pooling to the optimized ESP-NN kernel through the existing NHWC layout flow. Register the operator, provide its integer reference, and cover floor-mode pooling, padding without included pad values, batches, rounding, large windows, and portable fallback through shared dialect and device cases. Keep non-square outputs with multiple rows portable to avoid ESP-NN 1.4.1's output-row-stride bug. Enforce the restriction in the wrapper as well, and cover wide and tall outputs in both optimized channel paths. The upstream fix is released in 1.4.2; dependency adoption remains separate. Honor zero comparison tolerances in the ESP-IDF runner instead of silently using its defaults. Allow float32 epsilon only for pooling tests with portable sin branches. Validation passed 325 Python tests, 76 optimized ESP32-S3 emulator implementation cases across the pooling and binary suites, and scoped lint. Authored with OpenAI Codex.
1 parent 2b7d0d5 commit 9b4b2bc

13 files changed

Lines changed: 551 additions & 31 deletions

File tree

‎.ci/scripts/test_espressif.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ python examples/espressif/export_espnn_test_programs.py \
6868
espnn_ops=$(tr -d '\n' < "${build_dir}/espnn-tests/operators.txt" \
6969
| tr ',' '\n' | grep -v '^quantized_decomposed::' | paste -sd, -)
7070
if [[ "${espnn_kernels}" == optimized ]]; then
71-
espnn_ops="${espnn_ops},aten::sin.out,aten::relu.out,aten::max_pool2d_with_indices.out,aten::permute_copy.out"
71+
espnn_ops="${espnn_ops},aten::sin.out,aten::relu.out,aten::max_pool2d_with_indices.out,aten::avg_pool2d.out,aten::permute_copy.out"
7272
fi
7373

7474
cmake --preset esp-baremetal -G Ninja -B "${build_dir}/runtime" \

‎backends/espressif/CMakeLists.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ idf_component_register(
1010
"ops/op_quantized_mul_broadcast.cpp"
1111
"ops/op_quantized_add.cpp"
1212
"ops/op_quantized_max_pool2d.cpp"
13+
"ops/op_quantized_avg_pool2d.cpp"
1314
PRIV_REQUIRES
1415
esp-nn
1516
WHOLE_ARCHIVE

‎backends/espressif/README.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,15 @@ int8 tensor add with equal input shapes and multiply with equal shapes or a
2929
contiguous trailing block broadcast across the full tensor, including
3030
one-element tensors. It also accelerates values-only 4-D int8 MaxPool2d with matching input
3131
and output qparams, unit dilation, floor mode, and channel counts divisible by
32-
four. The shared Fused Quant layout pass introduces contiguous NHWC tensors for
32+
four. AvgPool2d uses the same shape and channel restrictions, with shared
33+
input/output qparams, floor mode, and no divisor override. Padding is supported
34+
when `count_include_pad=False`. AvgPool2d acceleration currently requires square
35+
outputs or a single output row because ESP-NN 1.4.1's optimized S3 kernel has
36+
an output-row-stride bug; other output shapes retain portable execution.
37+
The upstream fix is available in ESP-NN 1.4.2; this backend still pins 1.4.1.
38+
Its integer kernel rounds ties away from zero; the FQ reference rounds ties to
39+
even, so results can differ by one int8 step.
40+
The shared Fused Quant layout pass introduces contiguous NHWC tensors for
3341
spatial operators; model inputs and outputs retain their original layout.
3442
Required layout copies are visible in the operator-library report and accepted
3543
in strict mode. Unsupported pooling configurations retain portable execution.

‎backends/espressif/lowering.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,10 @@ def get_operator_library_report(edge: EdgeProgramManager) -> dict:
7777
if any(
7878
isinstance(neighbor.target, (EdgeOpOverload, OpOverload))
7979
and neighbor.target._schema.name
80-
== "esp_nn::quantized_max_pool2d_int8"
80+
in {
81+
"esp_nn::quantized_max_pool2d_int8",
82+
"esp_nn::quantized_avg_pool2d_int8",
83+
}
8184
for neighbor in neighbors
8285
):
8386
layout_ops.append(
Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
/*
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
* All rights reserved.
4+
*
5+
* This source code is licensed under the BSD-style license found in the
6+
* LICENSE file in the root directory of this source tree.
7+
*/
8+
9+
#include <executorch/runtime/kernel/kernel_includes.h>
10+
11+
#include <sdkconfig.h>
12+
13+
#include <esp_nn.h>
14+
15+
#include <algorithm>
16+
#include <cstdint>
17+
#include <limits>
18+
19+
namespace espressif::native {
20+
21+
using executorch::aten::IntArrayRef;
22+
using executorch::aten::Tensor;
23+
using executorch::runtime::KernelRuntimeContext;
24+
25+
// cppcheck-suppress unusedFunction
26+
Tensor& quantized_avg_pool2d_int8_out(
27+
KernelRuntimeContext& ctx,
28+
const Tensor& input,
29+
IntArrayRef kernel_size,
30+
IntArrayRef stride,
31+
IntArrayRef padding,
32+
Tensor& out) {
33+
ET_KERNEL_CHECK(
34+
ctx,
35+
input.dim() == 4 && out.dim() == 4 && kernel_size.size() == 2 &&
36+
stride.size() == 2 && padding.size() == 2,
37+
InvalidArgument,
38+
out);
39+
const auto batches = input.size(0);
40+
const auto height = input.size(1);
41+
const auto width = input.size(2);
42+
const auto channels = input.size(3);
43+
const auto output_height = out.size(1);
44+
const auto output_width = out.size(2);
45+
for (const auto value :
46+
{height, width, channels, output_height, output_width}) {
47+
ET_KERNEL_CHECK(
48+
ctx,
49+
value > 0 && value <= std::numeric_limits<uint16_t>::max(),
50+
InvalidArgument,
51+
out);
52+
}
53+
for (size_t axis = 0; axis < 2; ++axis) {
54+
ET_KERNEL_CHECK(
55+
ctx,
56+
kernel_size[axis] > 0 &&
57+
kernel_size[axis] <= std::numeric_limits<uint16_t>::max() &&
58+
stride[axis] > 0 &&
59+
stride[axis] <= std::numeric_limits<uint16_t>::max() &&
60+
padding[axis] >= 0 && padding[axis] <= kernel_size[axis] / 2,
61+
InvalidArgument,
62+
out);
63+
}
64+
const auto window_count = std::min<int64_t>(height, kernel_size[0]) *
65+
std::min<int64_t>(width, kernel_size[1]);
66+
ET_KERNEL_CHECK(
67+
ctx,
68+
128 * window_count + window_count / 2 <=
69+
std::numeric_limits<int32_t>::max(),
70+
InvalidArgument,
71+
out);
72+
const auto input_bytes = static_cast<uint64_t>(height) * width * channels;
73+
const auto output_bytes =
74+
static_cast<uint64_t>(output_height) * output_width * channels;
75+
ET_KERNEL_CHECK(
76+
ctx,
77+
channels % 4 == 0 && batches >= 0 &&
78+
(output_height == 1 || output_height == output_width) &&
79+
input_bytes <= std::numeric_limits<int32_t>::max() &&
80+
output_bytes <= std::numeric_limits<int32_t>::max() &&
81+
static_cast<size_t>(batches) <= input.nbytes() / input_bytes &&
82+
static_cast<size_t>(batches) <= out.nbytes() / output_bytes,
83+
InvalidArgument,
84+
out);
85+
for (int64_t batch = 0; batch < batches; ++batch) {
86+
esp_nn_avg_pool_s8(
87+
input.const_data_ptr<int8_t>() + batch * input_bytes,
88+
static_cast<uint16_t>(width),
89+
static_cast<uint16_t>(height),
90+
out.mutable_data_ptr<int8_t>() + batch * output_bytes,
91+
static_cast<uint16_t>(output_width),
92+
static_cast<uint16_t>(output_height),
93+
static_cast<uint16_t>(stride[1]),
94+
static_cast<uint16_t>(stride[0]),
95+
static_cast<uint16_t>(kernel_size[1]),
96+
static_cast<uint16_t>(kernel_size[0]),
97+
static_cast<uint16_t>(padding[1]),
98+
static_cast<uint16_t>(padding[0]),
99+
-128,
100+
127,
101+
static_cast<uint16_t>(channels));
102+
}
103+
return out;
104+
}
105+
106+
} // namespace espressif::native

‎backends/espressif/ops/operators.py‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,10 +49,19 @@
4949
"quantized_max_pool2d_int8.out(Tensor input, int[2] kernel_size, int[2] stride, "
5050
"int[2] padding, *, Tensor(a!) out) -> Tensor(a!)"
5151
)
52+
lib.define(
53+
"quantized_avg_pool2d_int8(Tensor input, int[2] kernel_size, int[2] stride, "
54+
"int[2] padding) -> Tensor"
55+
)
56+
lib.define(
57+
"quantized_avg_pool2d_int8.out(Tensor input, int[2] kernel_size, int[2] stride, "
58+
"int[2] padding, *, Tensor(a!) out) -> Tensor(a!)"
59+
)
5260

5361

5462
@register_fake("esp_nn::quantized_max_pool2d_int8")
55-
def quantized_max_pool2d_int8_meta(input, kernel_size, stride, padding):
63+
@register_fake("esp_nn::quantized_avg_pool2d_int8")
64+
def quantized_pool2d_int8_meta(input, kernel_size, stride, padding):
5665
batch, height, width, channels = input.shape
5766
output_height = (height + 2 * padding[0] - kernel_size[0]) // stride[0] + 1
5867
output_width = (width + 2 * padding[1] - kernel_size[1]) // stride[1] + 1
@@ -67,6 +76,28 @@ def quantized_max_pool2d_int8(input, kernel_size, stride, padding):
6776
return result.permute(0, 2, 3, 1).to(torch.int8).contiguous()
6877

6978

79+
@impl(lib, "quantized_avg_pool2d_int8", "CompositeExplicitAutograd")
80+
def quantized_avg_pool2d_int8(input, kernel_size, stride, padding):
81+
result = quantized_pool2d_int8_meta(input, kernel_size, stride, padding)
82+
for y in range(result.shape[1]):
83+
top = y * stride[0] - padding[0]
84+
bottom = min(top + kernel_size[0], input.shape[1])
85+
top = max(top, 0)
86+
for x in range(result.shape[2]):
87+
left = x * stride[1] - padding[1]
88+
right = min(left + kernel_size[1], input.shape[2])
89+
left = max(left, 0)
90+
count = (bottom - top) * (right - left)
91+
total = input[:, top:bottom, left:right, :].to(torch.int64).sum((1, 2))
92+
nudge = torch.where(total > 0, count // 2, -(count // 2))
93+
result[:, y, x, :] = (
94+
torch.div(total + nudge, count, rounding_mode="trunc")
95+
.clamp(-128, 127)
96+
.to(torch.int8)
97+
)
98+
return result
99+
100+
70101
@register_fake("esp_nn::quantized_mul_int8")
71102
def quantized_mul_int8_meta(
72103
input1,

‎backends/espressif/ops/operators.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,9 @@
2727
kernels:
2828
- arg_meta: null
2929
kernel_name: espressif::quantized_max_pool2d_int8_out
30+
31+
- func: esp_nn::quantized_avg_pool2d_int8.out(Tensor input, int[2] kernel_size, int[2] stride, int[2] padding, *, Tensor(a!) out) -> Tensor(a!)
32+
variants: function
33+
kernels:
34+
- arg_meta: null
35+
kernel_name: espressif::quantized_avg_pool2d_int8_out

‎backends/espressif/passes/lower_to_esp.py‎

Lines changed: 43 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -168,10 +168,9 @@ def _qparams(program, node, prefix):
168168
}
169169

170170

171-
def _pool_kernel_args(program, node) -> tuple | None:
171+
def _pool_kernel_args(program, node, output_value) -> tuple | None:
172172
inp = get_arg(node, "inp", torch.fx.Node)
173173
input_value = inp.meta.get("val")
174-
output_value = node.meta["val"][0]
175174
if not all(
176175
isinstance(value, torch.Tensor)
177176
and value.dim() == 4
@@ -182,7 +181,7 @@ def _pool_kernel_args(program, node) -> tuple | None:
182181
for value in (input_value, output_value)
183182
):
184183
return None
185-
# The optimized S3 kernel returns without writing for other channel counts.
184+
# Select the S3 assembly kernels, which require four-channel alignment.
186185
if input_value.shape[3] % 4:
187186
return None
188187
input_params = _qparams(program, node, "inp")
@@ -196,15 +195,15 @@ def _pool_kernel_args(program, node) -> tuple | None:
196195
kernel = get_arg(node, "kernel_size")
197196
stride = get_arg(node, "stride") or kernel
198197
pairs = []
199-
for arg in (kernel, stride, get_arg(node, "padding"), get_arg(node, "dilation")):
198+
for arg in (kernel, stride, get_arg(node, "padding")):
200199
if not isinstance(arg, (list, tuple)) or len(arg) not in (1, 2):
201200
return None
202201
pair = list(arg) * (2 if len(arg) == 1 else 1)
203202
if any(type(dim) is not int or not 0 <= dim <= 65535 for dim in pair):
204203
return None
205204
pairs.append(pair)
206-
kernel, stride, padding, dilation = pairs
207-
if get_arg(node, "ceil_mode") or dilation != [1, 1] or min(*kernel, *stride) == 0:
205+
kernel, stride, padding = pairs
206+
if get_arg(node, "ceil_mode") or min(*kernel, *stride) == 0:
208207
return None
209208
if any(pad > window // 2 for pad, window in zip(padding, kernel)):
210209
return None
@@ -224,12 +223,14 @@ def _pool_kernel_args(program, node) -> tuple | None:
224223

225224

226225
def _lower_max_pool2d(program, node) -> bool:
226+
if get_arg(node, "dilation") not in ([1], [1, 1], (1,), (1, 1)):
227+
return False
227228
values = list(node.users)
228229
if not values or any(
229230
user.target != operator.getitem or user.args[1] != 0 for user in values
230231
):
231232
return False
232-
args = _pool_kernel_args(program, node)
233+
args = _pool_kernel_args(program, node, node.meta["val"][0])
233234
if args is None:
234235
return False
235236
graph = node.graph
@@ -246,6 +247,35 @@ def _lower_max_pool2d(program, node) -> bool:
246247
return True
247248

248249

250+
def _lower_avg_pool2d(program, node) -> bool:
251+
if get_arg(node, "divisor_override") is not None:
252+
return False
253+
args = _pool_kernel_args(program, node, node.meta["val"])
254+
if args is None:
255+
return False
256+
output_height, output_width = node.meta["val"].shape[1:3]
257+
# ESP-NN 1.4.1's S3 assembly uses output height for the output row stride.
258+
if output_height != 1 and output_height != output_width:
259+
return False
260+
inp, kernel, _, padding = args
261+
if get_arg(node, "count_include_pad") and any(padding):
262+
return False
263+
height, width = inp.meta["val"].shape[1:3]
264+
count = min(height, kernel[0]) * min(width, kernel[1])
265+
# The vendor adds a signed rounding nudge to the int32 sum.
266+
if 128 * count + count // 2 > torch.iinfo(torch.int32).max:
267+
return False
268+
graph = node.graph
269+
with graph.inserting_before(node):
270+
replacement = graph.call_function(
271+
exir_ops.edge.esp_nn.quantized_avg_pool2d_int8.default, args
272+
)
273+
replacement.meta = node.meta.copy()
274+
node.replace_all_uses_with(replacement)
275+
graph.erase_node(node)
276+
return True
277+
278+
249279
class LowerToEsp(ExportedProgramPassBase):
250280
"""Lower supported fused-quant operators to ESP-NN."""
251281

@@ -255,6 +285,12 @@ def call(self, exported_program):
255285

256286
modified = False
257287
for node in list(graph.nodes):
288+
if (
289+
node.target
290+
== exir_ops.edge.fused_quant.avg_pool2d_channels_last.default
291+
):
292+
modified |= _lower_avg_pool2d(exported_program, node)
293+
continue
258294
if (
259295
node.target
260296
== exir_ops.edge.fused_quant.max_pool2d_with_indices_channels_last.default

‎backends/espressif/quantizer/esp_fused_quantizer.py‎

Lines changed: 26 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,30 +4,51 @@
44
# This source code is licensed under the BSD-style license found in the
55
# LICENSE file in the root directory of this source tree.
66

7+
import torch
8+
from executorch.backends.transforms.permute_pass_utils import get_arg
79
from executorch.backends.xnnpack.quantizer.xnnpack_quantizer import (
810
get_symmetric_quantization_config,
911
)
1012
from executorch.frontends.fused_quant.quantizer.quantizer import (
1113
FusedQuantQuantizer,
1214
get_core_aten_quantizers,
15+
OpQuantizer,
1316
)
17+
from torchao.quantization.pt2e.quantizer import SharedQuantizationSpec
18+
from torchao.quantization.pt2e.quantizer.utils import annotate_output_qspec
19+
20+
21+
class _AvgPoolQuantizer(OpQuantizer):
22+
def annotate_node(self, node: torch.fx.Node) -> None:
23+
super().annotate_node(node)
24+
annotate_output_qspec(node, SharedQuantizationSpec((node.args[0], node)))
1425

1526

1627
class _EspFusedQuantizer(FusedQuantQuantizer):
1728
"""Fused Quant quantizer for the Espressif operator-library backend.
1829
1930
Uses int8 per-tensor affine activations and symmetric weights, with
20-
per-channel weights for convolution. EspPassManager lowers supported
31+
per-channel weights for convolution. AvgPool2d without divisor_override
32+
shares input and output qparams. EspPassManager lowers supported
2133
Fused Quant operators to ESP-NN and decomposes the rest for portable
2234
execution. Use EspNNQuantizer for the EspNNBackend delegate instead.
2335
"""
2436

2537
def __init__(self) -> None:
38+
config = get_symmetric_quantization_config(is_per_channel=False)
2639
super().__init__(
27-
get_core_aten_quantizers(
28-
get_symmetric_quantization_config(is_per_channel=False),
29-
get_symmetric_quantization_config(is_per_channel=True),
30-
)
40+
[
41+
_AvgPoolQuantizer(
42+
torch.ops.aten.avg_pool2d.default,
43+
torch.ops.fused_quant.avg_pool2d.default,
44+
config,
45+
node_filter=lambda node: get_arg(node, "divisor_override") is None,
46+
),
47+
*get_core_aten_quantizers(
48+
config,
49+
get_symmetric_quantization_config(is_per_channel=True),
50+
),
51+
]
3152
)
3253

3354

0 commit comments

Comments
 (0)