Hello everyone! I just started learning about Deep Learning and artificial neural networks. To truly understand how these concepts work beyond the theory, I decided to build a complete Edge AI project from scratch.

My project is a real time digit recognition system. It takes numbers drawn on a touch screen and evaluates them live using a Convolutional Neural Network running locally on an ESP32 P4 microcontroller.

I learned a massive amount while figuring out the pipeline from PyTorch to C++, and I hope sharing my experiences and code will help others in the community avoid common pitfalls when deploying models to embedded hardware.

What is this project about?

The core idea is to bridge the gap between high level Python training and low level C++ execution.

While the standard MNIST dataset (containing 60,000 images of handwritten digits) is a great starting point, I quickly ran into a common Machine Learning problem: Dataset Bias. The MNIST digits are mostly American. For example, an American "1" is a single vertical line, while a European "1" has a distinct upstroke. An American "7" has no crossbar, but the European version does.

Because of this bias, the network would misclassify my European handwriting. To fix this, I collected custom handwriting samples from friends using a tablet and merged them with the MNIST dataset.

How does it work?

The project is split into two main phases:

1. The PyTorch Training (PC) I designed a Convolutional Neural Network in Python using PyTorch. To make sure the custom images do not get lost among the 60,000 MNIST images, I used Oversampling to multiply their weight. I also applied Data Augmentation (random rotation and translation) to force the model to learn the actual shapes instead of memorizing exact pixel locations.

2. The ESP DL Inference (Hardware) A microcontroller cannot efficiently run a 32 bit floating point Python model. Therefore, I used the ESP PPQ framework to compress the model into 8 bit integers (Int8). This quantization allows the ESP32 P4 to fit 16 network parameters into a single 128 bit register, executing multiple calculations in a single clock cycle.

The CrowPanel Advance 7" ESP32-P4 display shows the live output as a bar chart. You can toggle between a Softmax function (which forces a clear decision) and a raw scaling mode (which shows the actual confidence and uncertainty of the network).

Step by Step Tutorial

Step 1: Prepare the Environment Clone the repository and install the required Python packages (PyTorch, torchvision, onnxscript, and esp_ppq). Make sure your ESP IDF framework is set up for the hardware compilation.

Step 2: Train the Model Add your own 28x28 grayscale images to the custom_data folder if you want to teach the network your specific handwriting. Run the train_mnist_esp32.py script. The script normalizes the data, trains the network, and evaluates the accuracy.

Step 3: Quantization The Python script automatically handles the export. It converts the PyTorch model into an ONNX format and immediately uses ESP PPQ to quantize it into a highly compressed .espdl file.

Step 4: Deploy to the ESP32 P4 Copy the generated .espdl file from the output folder into your ESP32 C++ project directory. Compile the project using the ESP IDF and flash it to your board.

Step 5: Test and Tweak Start drawing on the touch screen! If the network struggles, check your drawing size. The model expects the digit to be centered and relatively large, just like the original MNIST dataset.

You can find a detailed tutorial on https://www.haraldkreuzer.net/en/news/esp32-p4-deep-learning-tutorial-mnist-pytorch-and-esp-dl