Skip to content

[Espressif] Add MLPerf Tiny operator-library coverage tests - #25265

Draft
rascani wants to merge 2 commits into
codex/espressif-fused-quant-conv-activation-fusionfrom
codex/espressif-mlperf-tiny-coverage
Draft

rascani wants to merge 2 commits into
codex/espressif-fused-quant-conv-activation-fusionfrom
codex/espressif-mlperf-tiny-coverage

Conversation

@rascani

@rascani rascani commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

The ESP operator-library tests now cover four complete MLPerf Tiny models: deep autoencoder, DS-CNN, ResNet-8 and MobileNetV1 0.25. The tests reuse the existing model definitions, including softmax outputs, with deterministic synthetic weights, inference inputs and calibration samples.

Each case checks the accelerated and portable operator inventory, compares lowering against the Fused Quant reference, serializes without delegates or residual FQ operators, and reports Q/DQ and layout nodes. The optimized ESP32-S3 CI job runs the same cases through the existing emulator runner and includes the required portable padding, mean and softmax kernels. It retains the clone registration needed by existing tests that return constant tensors.

Model ESP calls Portable compute Q/DQ Layout nodes
Deep autoencoder 10 0 2 0
DS-CNN 10 3 4 4
ResNet-8 13 4 8 2
MobileNetV1 0.25 28 7 12 2

The remaining portable compute is padding, pooling or mean, and softmax. Layout counts include views that serialization removes from runtime calls. The tests measure compiler/runtime correctness and coverage; synthetic fixtures do not measure trained-model accuracy.

Validation: all 554 operator, pass-manager, and model host tests pass, together with lintrunner and shell syntax checks. The Conv2d float_users implementation case passes on the ESP32-S3 emulator with the complete CI operator selection, confirming the restored clone registration. Local validation uses PyTorch 2.13.0+cpu and TorchAO 0.19.0.dev20261118+cpu; validation with the repository's PyTorch 2.14 pin remains for CI.

Classifier comparisons use an absolute tolerance of 1e-3 in probability space, which rejects injected 32-step classifier errors while accepting normal quantized rounding. The autoencoder retains its quantization tolerance. Earlier runs with a smaller local operator selection passed all four model implementation tests with this bound.

Model execution with the complete CI runtime is currently blocked by the example's 2 MiB app partition: the deep-autoencoder firmware is 2,222,048 bytes. Defer a partition change here while #25287 introduces larger 2944 KiB OTA app slots, then re-run the model implementation tests with that configuration.

Authored with OpenAI Codex.

@rascani
rascani added this pull request to stack #25262 November 19, 2026 02:37
@pytorch-bot

pytorch-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25265

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Cancelled Jobs, 3 Pending

As of commit dbc6f54 with merge base 351e938 (image):

NEW FAILURES - The following jobs have failed:

CANCELLED JOBS - The following jobs were cancelled. Please retry:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 19, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@rascani
rascani force-pushed the codex/espressif-mlperf-tiny-coverage branch from 43c96cf to fe97ef3 Compare November 19, 2026 16:58
@rascani
rascani force-pushed the codex/espressif-mlperf-tiny-coverage branch from fe97ef3 to 31891a0 Compare November 20, 2026 18:13
@rascani
rascani force-pushed the codex/espressif-mlperf-tiny-coverage branch from aef7aad to 5c1649e Compare November 20, 2026 23:00
@rascani
rascani force-pushed the codex/espressif-mlperf-tiny-coverage branch from 5c1649e to 64bd749 Compare November 21, 2026 00:02
Exercise deep autoencoder, DS-CNN, ResNet-8 and MobileNetV1 0.25 through EspTester using deterministic weights, inputs and calibration. Assert accelerated and portable operator inventories, compare lowering with the Fused Quant reference, check serialization, and print Q/DQ and layout counts.

Run the same cases on the ESP32-S3 emulator in the optimized CI job. Include the portable padding, mean and softmax kernels needed by the models.

Compare classifier probabilities with an absolute tolerance of 1e-3 instead of using intermediate activation scales. Retain the autoencoder quantization tolerance.

Authored with OpenAI Codex.
Keep aten::clone.out available in the optimized ESP32-S3 test runtime. The existing Conv2d float_users case returns a constant bias, which requires the clone inserted by constant folding when the program is serialized. Removing the registration in the model coverage change prevented that method from loading.

The manual list remains until the deferred PTE tooling can derive the union of required operators from the test programs. Shell syntax and scoped lintrunner pass.

Authored with OpenAI Codex.
@rascani
rascani force-pushed the codex/espressif-mlperf-tiny-coverage branch from 64bd749 to dbc6f54 Compare November 21, 2026 02:05

This branch was previously deployed

1 inactive deployment
cadence — dbc6f542 Deployed Nov 21, 2026 by rascani via hifi5s-op-test / hifi5s #39860
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant