Skip to content

Commit 95a06a1

Browse files
committed
[Espressif] Validate convolution scratch query dimensions
Reject convolution and depthwise convolution dimensions, strides, and channel multipliers outside [1, 65535], and padding outside [0, 65535], before calling ESP-NN scratch-size queries. Vendor dispatch narrows these values to uint16; a zero or wrapped filter width can otherwise terminate Python with a divide-by-zero instead of raising an exception. Document the bounds and exercise invalid values and valid boundaries across the S3, RISC-V PIE, generic, and ANSI bindings. The sizing-only binding build, 161 scratch tests, and lintrunner passed. Authored with OpenAI Codex.
1 parent ea4fb9a commit 95a06a1

7 files changed

Lines changed: 369 additions & 23 deletions

File tree

‎.ci/scripts/test_espressif.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -160,7 +160,8 @@ if [[ "${espnn_kernels}" == optimized ]]; then
160160
-DESP_NN_LOCAL_PATH="${PWD}/examples/espressif/project/managed_components/espressif__esp-nn" \
161161
-DCMAKE_LIBRARY_OUTPUT_DIRECTORY="${esp_python_dir}"
162162
logged "scratch-build.log" cmake --build "${build_dir}/scratch-sizing"
163-
logged "scratch-tests.log" python -m pytest -q backends/espressif/test/test_scratch.py
163+
logged "scratch-tests.log" env EXECUTORCH_REQUIRE_ESP_NN_SCRATCH=1 \
164+
python -m pytest -q backends/espressif/test/test_scratch.py
164165
# Stop on failure so later cases cannot overwrite its firmware and logs.
165166
ESPRESSIF_BUILD_DIR="${build_dir}" python -m pytest \
166167
-x -v --tb=short backends/espressif/test/ops -k implementation \

‎.github/workflows/test-espressif.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,10 @@ on:
2626
# Read for the torch the component's VERSION and README name.
2727
- install_requirements.py
2828
- install_utils.py
29+
# Build and package the optional ESP-NN scratch bindings.
30+
- setup.py
31+
- CMakeLists.txt
32+
- tools/cmake/preset/default.cmake
2933
# Exported by the consumer test.
3034
- examples/models/mlperf_tiny/**
3135
- devtools/bundled_program/**

‎backends/espressif/README.md‎

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,10 @@ CMAKE_ARGS="-DEXECUTORCH_BUILD_ESP_NN_PYBINDS=ON" pip install -e . --no-build-is
6363

6464
The host libraries retain only sizing code. CMake fetches the pinned ESP-NN
6565
sources; `ESP_NN_LOCAL_PATH` can reuse an existing 1.5.0 checkout or ESP-IDF
66-
managed component.
66+
managed component. The host build uses GNU-style compiler section flags and
67+
linker garbage collection (`--gc-sections`, or `-dead_strip` on Apple platforms);
68+
MSVC-style toolchains fail at configuration. Downloaded sources are checked
69+
against the pinned SHA-256, and a local source tree must declare version 1.5.0.
6770

6871
Select the device's `EspTarget` and `EspKernelMode` to match its ESP-NN build:
6972

@@ -82,15 +85,26 @@ if queries is not None:
8285
)
8386
```
8487

85-
Dimensions use `(width, height, channels)`; stride and padding use
86-
`(width, height)`. Convolution queries use unit dilation. Sizes are in bytes,
87-
including zero for kernels that need no scratch. `get_scratch_queries` returns
88-
`None` when the selected binding is unavailable.
88+
Convolution queries use `(width, height, channels)` for dimensions,
89+
`(width, height)` for stride and padding, and unit dilation. Dimensions,
90+
strides, and depthwise channel multipliers must be in `[1, 65535]`; padding must be in
91+
`[0, 65535]`. Conservative bounds on padded tensors, aligned filters, staging,
92+
and coordinates reject shapes that could overflow the vendor's int32 arithmetic
93+
before calling the sizing query. These bounds also apply to the fully connected
94+
query's `row_len` and `out_ch`. Softmax and mean dimensions, and fully connected
95+
`batches`, must be positive int32 values; softmax width must also fit an int32
96+
byte count for four bytes per element. Invalid bounds raise `ValueError`.
97+
Sizes are in bytes, including zero for kernels that need no scratch. The vendor
98+
fully connected query also returns zero for batches above 65535.
99+
`get_scratch_queries` returns `None` only when the selected module is absent;
100+
errors loading an installed binding propagate to the caller.
89101

90102
Optimized mode selects S3 queries for ESP32-S3, shared PIE queries for
91103
ESP32-P4/ESP32-S31, and generic queries for other `EspTarget` values. ANSI mode
92104
selects ANSI queries for every chip. Query results are checked on the host for
93105
all configurations; device execution tests currently cover ESP32-S3.
106+
CI sets `EXECUTORCH_REQUIRE_ESP_NN_SCRATCH=1` when running `test_scratch.py`, so
107+
all four profiles and all seven query families must be available and pass.
94108

95109
## Contents
96110

‎backends/espressif/scratch/CMakeLists.txt‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,18 +9,44 @@ if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
99
project(esp_nn_scratch LANGUAGES C CXX)
1010
endif()
1111

12+
if(MSVC
13+
OR CMAKE_C_COMPILER_FRONTEND_VARIANT STREQUAL "MSVC"
14+
OR CMAKE_CXX_COMPILER_FRONTEND_VARIANT STREQUAL "MSVC"
15+
)
16+
message(
17+
FATAL_ERROR
18+
"ESP-NN scratch bindings require GNU-style section flags; MSVC-style toolchains are unsupported."
19+
)
20+
endif()
21+
1222
set(ESP_NN_LOCAL_PATH
1323
""
1424
CACHE PATH "ESP-NN 1.5.0 sources for scratch sizing"
1525
)
1626
if(ESP_NN_LOCAL_PATH)
27+
if(NOT EXISTS "${ESP_NN_LOCAL_PATH}/include/esp_nn.h"
28+
OR NOT EXISTS "${ESP_NN_LOCAL_PATH}/idf_component.yml"
29+
)
30+
message(
31+
FATAL_ERROR "ESP_NN_LOCAL_PATH must name an ESP-NN 1.5.0 source tree."
32+
)
33+
endif()
34+
file(STRINGS "${ESP_NN_LOCAL_PATH}/idf_component.yml" esp_nn_version
35+
REGEX "^version:[ \t]*['\"]?1\\.5\\.0['\"]?[ \t]*$"
36+
)
37+
if(NOT esp_nn_version)
38+
message(FATAL_ERROR "ESP_NN_LOCAL_PATH must contain ESP-NN version 1.5.0.")
39+
endif()
1740
set(esp_nn_SOURCE_DIR "${ESP_NN_LOCAL_PATH}")
1841
else()
1942
include(FetchContent)
2043
FetchContent_Declare(
2144
esp_nn
2245
URL https://github.com/espressif/esp-nn/archive/refs/tags/v1.5.0.tar.gz
23-
SOURCE_SUBDIR src/convolution
46+
URL_HASH
47+
SHA256=63e8432b00c354eecb7dfab7133c47bc50023337cd6ea1c4dec67ca1060fef17
48+
# Populate sources without adding ESP-NN's ESP-IDF component CMake project.
49+
SOURCE_SUBDIR src/convolution
2450
)
2551
FetchContent_MakeAvailable(esp_nn)
2652
endif()

‎backends/espressif/scratch/__init__.py‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,10 @@ def get_scratch_queries(
2424
EspTarget.ESP32P4: "riscv_pie",
2525
EspTarget.ESP32S31: "riscv_pie",
2626
}.get(target, "generic")
27+
module_name = f"executorch.backends.espressif._esp_nn_scratch_{profile}"
2728
try:
28-
return importlib.import_module(
29-
f"executorch.backends.espressif._esp_nn_scratch_{profile}"
30-
)
31-
except ImportError:
32-
return None
29+
return importlib.import_module(module_name)
30+
except ModuleNotFoundError as error:
31+
if error.name == module_name:
32+
return None
33+
raise

‎backends/espressif/scratch/bindings.cpp‎

Lines changed: 127 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,21 +6,96 @@
66
* LICENSE file in the root directory of this source tree.
77
*/
88

9+
#include <algorithm>
910
#include <array>
11+
#include <cstddef>
1012
#include <cstdint>
13+
#include <initializer_list>
14+
#include <limits>
15+
#include <string>
1116

1217
#include <esp_nn.h>
1318
#include <pybind11/pybind11.h>
1419
#include <pybind11/stl.h>
1520

1621
namespace {
1722

23+
template <std::size_t N>
24+
void check_bounds(
25+
const std::array<int32_t, N>& values,
26+
const char* name,
27+
int32_t minimum = 1,
28+
int32_t maximum = std::numeric_limits<uint16_t>::max()) {
29+
if (std::any_of(values.begin(), values.end(), [=](int32_t value) {
30+
return value < minimum || value > maximum;
31+
})) {
32+
throw pybind11::value_error(
33+
std::string(name) + " must be in [" + std::to_string(minimum) + ", " +
34+
std::to_string(maximum) + "]");
35+
}
36+
}
37+
38+
int64_t checked_product(std::initializer_list<int64_t> factors) {
39+
int64_t product = 1;
40+
for (const auto factor : factors) {
41+
if (factor > std::numeric_limits<int32_t>::max() / product) {
42+
throw pybind11::value_error(
43+
"scratch query intermediates exceed the int32 range");
44+
}
45+
product *= factor;
46+
}
47+
return product;
48+
}
49+
50+
void check_conv_arithmetic(
51+
const std::array<int32_t, 3>& input,
52+
const std::array<int32_t, 3>& filter,
53+
const std::array<int32_t, 3>& output,
54+
const std::array<int32_t, 2>& stride,
55+
const std::array<int32_t, 2>& padding,
56+
int32_t channel_multiplier = 1) {
57+
const int64_t aligned_channels = (int64_t{input[2]} + 15) & ~int64_t{15};
58+
const int64_t output_channels =
59+
std::max(int64_t{output[2]}, int64_t{input[2]} * channel_multiplier);
60+
std::array<int64_t, 2> extent;
61+
for (std::size_t axis = 0; axis < extent.size(); ++axis) {
62+
extent[axis] =
63+
std::max(int64_t{input[axis]}, int64_t{output[axis]} * stride[axis]) +
64+
2 * int64_t{padding[axis]} + filter[axis];
65+
}
66+
// Bound padded copies, aligned filters and coordinates before vendor int32
67+
// arithmetic. S3 stages 16 windows plus 24 windows of panel work and a 16 KiB
68+
// panel. Four int32 channel arrays and 256 bytes cover parameters and
69+
// alignment.
70+
const auto input_bound =
71+
checked_product({extent[0], extent[1], aligned_channels});
72+
const auto filter_bound = checked_product(
73+
{int64_t{filter[0]} * aligned_channels + 15, filter[1], output_channels});
74+
const auto output_bound = checked_product(
75+
{output[0], output[1], std::max(aligned_channels, output_channels)});
76+
const auto window_bound =
77+
checked_product({filter[0], filter[1], aligned_channels});
78+
const auto bound = 2 * input_bound + 2 * filter_bound + output_bound +
79+
40 * window_bound + 16 * std::max(aligned_channels, output_channels) +
80+
16 * 1024 + 256;
81+
if (bound > std::numeric_limits<int32_t>::max()) {
82+
throw pybind11::value_error(
83+
"scratch query intermediates exceed the int32 range");
84+
}
85+
}
86+
1887
int conv2d_scratch_size(
1988
std::array<int32_t, 3> input_dims,
2089
std::array<int32_t, 3> filter_dims,
2190
std::array<int32_t, 3> output_dims,
2291
std::array<int32_t, 2> stride,
2392
std::array<int32_t, 2> padding) {
93+
check_bounds(input_dims, "input_dims");
94+
check_bounds(filter_dims, "filter_dims");
95+
check_bounds(output_dims, "output_dims");
96+
check_bounds(stride, "stride");
97+
check_bounds(padding, "padding", 0);
98+
check_conv_arithmetic(input_dims, filter_dims, output_dims, stride, padding);
2499
const data_dims_t input{input_dims[0], input_dims[1], input_dims[2], 0};
25100
const data_dims_t filter{filter_dims[0], filter_dims[1], filter_dims[2], 0};
26101
const data_dims_t output{output_dims[0], output_dims[1], output_dims[2], 0};
@@ -38,6 +113,19 @@ int depthwise_conv2d_scratch_size(
38113
std::array<int32_t, 2> stride,
39114
std::array<int32_t, 2> padding,
40115
int32_t channel_multiplier) {
116+
check_bounds(input_dims, "input_dims");
117+
check_bounds(filter_dims, "filter_dims");
118+
check_bounds(output_dims, "output_dims");
119+
check_bounds(stride, "stride");
120+
check_bounds(padding, "padding", 0);
121+
check_bounds(std::array{channel_multiplier}, "channel_multiplier");
122+
check_conv_arithmetic(
123+
input_dims,
124+
filter_dims,
125+
output_dims,
126+
stride,
127+
padding,
128+
channel_multiplier);
41129
const data_dims_t input{input_dims[0], input_dims[1], input_dims[2], 0};
42130
const data_dims_t filter{filter_dims[0], filter_dims[1], filter_dims[2], 0};
43131
const data_dims_t output{output_dims[0], output_dims[1], output_dims[2], 0};
@@ -50,6 +138,42 @@ int depthwise_conv2d_scratch_size(
50138
&input, &filter, &output, &params);
51139
}
52140

141+
int32_t softmax_scratch_size(int32_t width, int32_t height) {
142+
check_bounds(
143+
std::array{width, height},
144+
"softmax dimensions",
145+
1,
146+
std::numeric_limits<int32_t>::max());
147+
checked_product({width, sizeof(int32_t)});
148+
return esp_nn_get_softmax_scratch_size(width, height);
149+
}
150+
151+
int32_t mean_scratch_size(int32_t height, int32_t width, int32_t channels) {
152+
check_bounds(
153+
std::array{height, width, channels},
154+
"mean dimensions",
155+
1,
156+
std::numeric_limits<int32_t>::max());
157+
return esp_nn_get_mean_scratch_size(height, width, channels);
158+
}
159+
160+
int fc_batch_scratch_size(int32_t batches, int32_t row_len, int32_t out_ch) {
161+
check_bounds(
162+
std::array{batches}, "batches", 1, std::numeric_limits<int32_t>::max());
163+
check_bounds(std::array{row_len}, "row_len");
164+
check_bounds(std::array{out_ch}, "out_ch");
165+
if (batches <= std::numeric_limits<uint16_t>::max()) {
166+
check_conv_arithmetic(
167+
{1, batches, row_len},
168+
{1, 1, row_len},
169+
{1, batches, out_ch},
170+
{1, 1},
171+
{0, 0});
172+
}
173+
return esp_nn_get_fc_batch_scratch_size(
174+
batches, static_cast<uint16_t>(row_len), static_cast<uint16_t>(out_ch));
175+
}
176+
53177
} // namespace
54178

55179
namespace py = pybind11;
@@ -77,12 +201,12 @@ PYBIND11_MODULE(ESP_NN_SCRATCH_MODULE, m) {
77201
py::arg("channel_multiplier"));
78202
m.def(
79203
"softmax_scratch_size",
80-
&esp_nn_get_softmax_scratch_size,
204+
&softmax_scratch_size,
81205
py::arg("width"),
82206
py::arg("height"));
83207
m.def(
84208
"mean_scratch_size",
85-
&esp_nn_get_mean_scratch_size,
209+
&mean_scratch_size,
86210
py::arg("height"),
87211
py::arg("width"),
88212
py::arg("channels"));
@@ -91,7 +215,7 @@ PYBIND11_MODULE(ESP_NN_SCRATCH_MODULE, m) {
91215
});
92216
m.def(
93217
"fc_batch_scratch_size",
94-
&esp_nn_get_fc_batch_scratch_size,
218+
&fc_batch_scratch_size,
95219
py::arg("batches"),
96220
py::arg("row_len"),
97221
py::arg("out_ch"));

0 commit comments

Comments
 (0)