Experiments exploring neural network performance on the pico2
gen_cnn.py is a simple script to generate Conv2D models. It allows exploration of the performance of a compute bound workload on Pico2. The script accepts a single parameter which is the number of layers. In this way performance of the defined network can be scaled up linearly. Performance of a single layer is controlled by varying the input tensor shape and convolution parameters. Padding is required since the output tensor and input and output tensor shapes must be the same so the layers will stack properly. The number of multiply-accumulate operations in the model is embedded in the output model file name.
gen_fc.py is a simple script to generate FullyConnected models. It allows exploration of the performance of a memory bandwidth bound workload on Pico2. The script accepts two parameters: the number of layers and the width. In this way required computation and memory bandwidth of the defined network can be scaled up linearly. FullyConnected layers are inserted in pairs (P→Q, Q→P) where P is the number of inputs and Q is the width. This allows finer grained control of compute and memory. The number of multiply-accumulate operations in the model is embedded in the output model file name.
convert_model.py is a simple script to convert the ONNX models created by the model generators into C++ code that can be included in the profiling application (main_function.cpp). This script depends on onnx2tf which has not been updated in a while (and requires an older version Tensorflow). It is recommended to create a Python virtual environment for onnx2tf and use that when running convert_model.py. The conversion process leaves behind TFlite models which can be inspected to see which TFLM operations are being used (e.g., using Netron).
main_function.cpp is simply a hacked version of the pico-tflmicro Hello World example. In addition to pico-tflmicro, it also depends on the pico-sdk and the pico-sdk-tools.
Follow the respective instructions to download and build these projects first. Then just replace the code in pico-tflmicro/examples/hello_world with this code and rebuild.
It is necessary to replace the contents of hello_world_int8_model_data.cpp and hello_world_float_model_data.cpp with that of the corresponding cpp files generated by convert_model.py. Don't forget to change the variable names to match those defined in the corresponding headers.
Some defines in main_functions.cpp are relevant:
- PROFILE_MODEL - turns on profiling (assumes profiling has been enabled when building the TFLM SDK)
- PROFILE_COUNT - how often to profile (profiling too often overwhelms the serial console)
- INT8_MODEL - uses the int8 neural network instead of the float32 one
- ARENA_SIZE - this is a complete swag at the runtime memory required for the model (thanks a lot, ARM)
The application repeatedly invokes the neural network, printing the performance to the serial console. Inference takes place as fast as the RP2350 can compute it so results are printed every PROFILE_COUNT inferences. The green LED on the RP2350 is toggled each time a result is printed to the serial console.
The TinyML benchmark provides a very informative way to compare performance across platforms and runtimes. Results from many different platforms are posted on the ML Commons website. The benchmark consists of five "tiny" models that represent practical use cases: Anomaly Detection, Image Classification, Keyword Spotting, Streaming Wake Word, and Visual Wake Word. The model architectures are FC Auto-Encoder, ResNet-V1, DSCNN, 1D DSCNN, and MobileNetV1, respectively. Pre-trained models are found in the training/<case>/trained_models folders.
convert_tinyml_models.py is a simple script to convert the h5 and pb models from the TinyML repository into C++ code that can be included in the profiling application (main_function.cpp). The conversion process leaves behind TFlite models which can be inspected to see which TFLM operations are being used (e.g., using Netron). For TinyML, only int8 models are created (fp32 models are straightforward but less interesting due to lower performance).
For TinyML, one version of main_function.cpp is provided per model. As before, it is a hacked version of the pico-tflmicro Hello World example. The same description and instructions as above apply to each one. Each of the provided main_function_*.cpp files will need to be renamed to main_function.cpp before compiling the application.
The above model conversion and deployment procedure was carried out using the Waveshare RP2350-Plus-16MB device. In the table below, the inference time is compared to the reported inference time on a comparable platform, the Nucleo NUCLEO-U575ZI-Q (as reported in the MLPerf v1.3 results).
| Platform | Clock | Model | Time(msec) | Speedup |
|---|---|---|---|---|
| Waveshare | 150MHz | Anomaly Detection | 58.95 | 1.00 |
| Nucleo | 160MHz | Anomaly Detection | 3.21 | 18.37 |
| Waveshare | 150MHz | Image Classification | 1709.93 | 1.00 |
| Nucleo | 160MHz | Image Classification | 109.49 | 15.62 |
| Waveshare | 150MHz | Keyword Spotting | 60.40 | 1.00 |
| Nucleo | 160MHz | Keyword Spotting | 28.99 | 2.08 |
| Waveshare | 150MHz | Streaming Wakeword | 192.41 | 1.00 |
| Nucleo | 160MHz | Streaming Wakeword | 11.66 | 16.51 |
| Waveshare | 150MHz | Visual Wakeword | 1375.02 | 1.00 |
| Nucleo | 160MHz | Visual Wakeword | 59.27 | 23.20 |
The difference in performance between the two platforms is quite significant, with the reported Nucleo performance being up to 23x faster than that of Waveshare. But it should be pointed out that the neural network deployment toolchains are different in the two cases. As described above, the open source pico-tflmicro package was used to prepare the models for the Waveshare device. The toolchain reportedly used for the Nucleo device is X-CUBE-AI v10.2 from STMicroelectronics.
After acquiring a NUCLEO-U575ZI-Q board, installing STM32Cube AI Studio, and investigating, an explanation for the performance gap emerged. First, it turns out that the pico-tflmicro example code stores the neural network in flash. Moving the MLPerf TinyML models entirely to SRAM is not possible in all cases since some of the models are too big. The streaming wake word model, along with a tiny proprietary model with FC, Tanh, ReLU, and Minimum layers were measured. The results are shown below.
| Platform | Clock | Model | Time(msec) | Speedup |
|---|---|---|---|---|
| Waveshare | 150MHz | Proprietary FP32 | 0.252 | 1.00 |
| Nucleo | 160MHz | Proprietary FP32 | 0.240 | 1.05 |
| Waveshare | 150MHz | Proprietary INT8 | 0.416 | 1.00 |
| Nucleo | 160MHz | Proprietary INT8 | 0.089 | 4.67 |
| Waveshare | 150MHz | Streaming Wakeword FP32 | 51.542 | 1.00 |
| Nucleo | 160MHz | Streaming Wakeword FP32 | 47.80 | 1.08 |
Taking the CPU clock difference into account, the FP32 performance is the same on the Waveshare and NUCLEO boards. The difference in INT8 performance is still quite large. Detailed profiling showed that the INT8 Tanh operator took 217 usec on average on the RP2350 while the FP32 took only 36 usec. On the NUCLEO-U575ZI-Q board, the INT8 Tanh operator took 4.6 usec while the FP32 one took 61 usec. Comparing the largest FC operator, the RP2350 performance was 93 usec (INT8) / 139 usec (FP32) while the NUCLEO-U575ZI-Q performance was 51 usec (INT8) / 141 usec (FP32).
The advantages of the STMicro NUCLEO-U575ZI-Q appear to be:
- Larger SRAM allows larger models to run more efficiently.
- Slightly (7%) higher clock speed.
- Better INT8 optimization of CMSIS-NN compared to generic ARM CMSIS-NN library.
This is the list of packages and their versions used in these experiments.
Package Version
---------------------------- --------
absl-py 2.3.1
astunparse 1.6.3
cachetools 5.5.2
certifi 2025.8.3
charset-normalizer 3.4.3
coloredlogs 15.0.1
flatbuffers 25.9.23
gast 0.4.0
google-auth 2.41.0
google-auth-oauthlib 1.0.0
google-pasta 0.2.0
grpcio 1.70.0
h5py 3.11.0
humanfriendly 10.0
idna 3.10
importlib_metadata 8.5.0
keras 2.13.1
libclang 18.1.1
Markdown 3.7
markdown-it-py 3.0.0
MarkupSafe 2.1.5
mdurl 0.1.2
mpmath 1.3.0
numpy 1.24.3
oauthlib 3.3.1
onnx 1.17.0
onnx_graphsurgeon 0.5.8
onnx2tf 1.20.0
onnxruntime 1.19.2
onnxsim 0.4.36
opt_einsum 3.4.0
packaging 25.0
pip 25.0.1
protobuf 4.25.8
psutil 7.1.0
pyasn1 0.6.1
pyasn1_modules 0.4.2
Pygments 2.19.2
requests 2.32.4
requests-oauthlib 2.0.0
rich 14.2.0
rsa 4.9.1
setuptools 56.0.0
six 1.17.0
sng4onnx 1.0.4
sympy 1.13.3
tensorboard 2.13.0
tensorboard-data-server 0.7.2
tensorflow 2.13.1
tensorflow-estimator 2.13.0
tensorflow-io-gcs-filesystem 0.34.0
termcolor 2.4.0
typing_extensions 4.5.0
urllib3 2.2.3
Werkzeug 3.0.6
wheel 0.45.1
wrapt 1.17.3
zipp 3.20.2