Introduction

This is Avnet's IoTConnect Image Classification Demo for STM23 MP2, with AWS SageMaker and OTA updates support.

This project demonstrates how a Tensorflow base model can be converted to TFLite format and uploaded to a device with IoTConnect OTA with an application that can read images from a MIPI CSI-2 or a USB camera and perform image classification inference on the device with a Mobilenet V2 model.

Project Features

This project contains the several demos:

• The application running on the device with:
  • Inferencing a camera feed on a MobileNet V2 model on the device with the STAI Python API.
  • Support for both USB and B-CAMS-IMX camera.
  • Reporting inference data to /IOTCONNECT.
  • AWS Kinesis Video Streams for video streaming to AWS.
  • Uploading the camera images to /IOTCONNECT AWS S3 bucket for later analysis and fine-tuning of the model.
  • Receiving OTA updates of the model from /IOTCONNECT.
  • Pre-made models provided for reference and ease of use for evaluation:
    • Reference ST ModelZoo quantized model
    • Modern MobileNet V2 model quantized with the provided quantization pipeline.
    • Sample MobileNet V2 model fine-tuned to recognize various STM32 development boards.
• The quantization pipeline:
  • Quantizes the base to TFLite per-tensor (default) or per-channel format.
  • Optimizes the input model using the ST's model optimization for better performance on the device.
  • Converts the model to NBG format with the stedgeai tool for best performance on the device.
  • Sends an OTA update of the model to the device with /IOCONNECT REST API.
  • Support for running the quantization on AWS SageMaker with S3 images delivered by the application running on the device.
• Example fine-tuning use case:
  • Fine-tunes the model to recognize new classes:
    • A generic "development board".
    • The STM32-MP135f-DK device.
    • The STM23-MP35F-EV1.
  • Provided image samples to support the new classes for easy evaluation.

Requirements

To run the demo:

• An STM32-MP257F-DK
• OpenSTLinux Starter Package A35-TD flavor 6.2.0 (6.6-yocto-scarthgap-mpu-v26.02.18)
• Display connected to the device via HDMI or LVDS.
• A CAMS-IMX MIPI CSI-2 camera connected to the device or a USB camera.

Development system requirements:

• Python 3.12. Use pyenv if you need to manage multiple versions of Python on your system.
• To be able to create NBG models on your development system, install [STE AI Core 2.2.0]https://www.st.com/en/development-tools/stedgeai-core.html
• To evaluate SageMaker and WebRTC with AWS, you will need to set up an AWS account and configure AWS CLI on your system.

Device Support

This project supports STM32MP257f-dk by default. To support other MP2 devices, a minor modification to the stedgeai-convert.sh script would be needed to invoke the correct target for the staidgeai tool.

While MP1 devices can be supported in theory, getting the python packages to install can be difficult due to limited availability of pre-compiled native python packages for armv7l (vs. the aarch64 for MP2). Therefore, MP1 support is not included in this project.

Device Setup

Image Flashing

Download the OpenSTLinux STM32MP2 OpenSTLinux Starter Package for A35-TD flavor (MP2-START-A35-TD), version 6.2.0 from https://wiki.st.com/stm32mpu/wiki/Category:OpenSTLinux_st

Follow the ST Instructions to flash the OpenSTLinux Starter Package image to your device at https://wiki.st.com/stm32mpu/wiki/Category:OpenSTLinux_starter_packages (Click "STM32MP25 Discovery kits - Starter Package" on the left side menu)

The instructions provided in this document are tested with the StarterPackage version 6.2.0. Keep in mind that once the package is downloaded, the actual version may differ. For example: 6.6-yocto-scarthgap-mpu-v26.02.18 was tested with STM32MP257-DK.

The overall process with STM32CubeProgrammer is fairly complex and can be lengthy. As an advanced but faster alternative, we suggest to explore the option of downloading the starter package, and running the create_sdcard_from_flashlayout.sh utility instead in the scripts directory of the package in order to create an SD card image. This SD card image can then be flashed onto the SD card with the dd utility, Rufus, Balena Etcher and similar on other OS-es.

Display Setup

Connect a display via an HDMI cable or use an appropriate LVDS display.

Camera Setup

It is recommended to use the B-CAMS-IMX MIPI CSI-2 camera for this demo.

Useful Read: STM32MP2 V4L2 camera overview

A USB camera may be used as well. See the section below for more details and troubleshooting the camera setup.

Software and Device Install Steps

Once the device has been flashed with the required image, connect the board's network connection and determine the IP address of your device using your Wi-Fi router's client list, or by connecting to the USB console with a terminal emulator program and typing ifconfig, or by querying devices on your subnet with nmap on Linux - for example if your local subnet is 192.168.38.* you may get an output like this:

nmap -sP 192.168.38.* | grep -v latency
...
Nmap scan report for stm32mp2-e3-d4-fc (192.168.38.141)
...

The device IP will be used in next steps.

Prepare The Application

An installer script is provided with the application that simplifies the installation for your MP2 device. In order to prepare for running the script, first copy the application into the app directory on the device (example device IP being 192.168.38.141):

device_ip=192.168.38.141
scp -rp app root@$device_ip:

Set up The /IOTCONNECT Account

Import a new /IOTCONNECT template in your /IOTCONNECT account using app/staiicdemo-device-template.json

Create a new device using an autogenerated certificate in your account and download the certificate into the root of this repo. The certificate can be downloaded navigating to Devices -> Device -> YourDevice -> Info Panel (on the left) -> connection info (link on the right) and then click the orange certificate with a green download arrow.

Download the iotConfigJson.json file into the root of this repo by clicking the Paper-and-cog icon above the connection info link.

Upload the files into the app directory on the device with SCP:

scp *certificates.zip  root@$device_ip:app/
scp iotcDeviceConfig.json  root@$device_ip:app/

SSH to the device and execute the installer:

cd ~/app
./device-setup.sh <your device *certificates.zip>

The setup script will create a virtual environment with necessary packages installed at .venv-staiicdemo in user's home. It will also download pre-quantized reference model from ST:

• mobilenet_v2_1.0_224_int8_per_tensor.nb: Network Binary Graph (NBG) model.
• mobilenet_v2_1.0_224_int8_per_tensor.tflite: TFLite model.

Pre-quantized and pre-fine-tuned models are provided as well:

• mobilenetv2-finetuned.nb and .tflite - fine-tuned to recognize various STM32 development boards

The sample application is based on How to run inference using the STAI MPU Python API example on ST wiki

Running The Application

To launch the application on the device, connect with SSH and run these commands at the prompt:

~/app/app.sh -m mobilenet_v2_1.0_224_int8_per_tensor.nb

Or choose one of the other models that were downloaded during the setup step.

Once the application runs a model, the latest running model file path will be recorded into the model-name.txt file. This model file will be read from the same file if the application does not take the --model-file command line argument. When a new model has been downloaded with OTA in later steps, if you need to re-run the application you can simply run the app without the model argument.

python3 app.py

/IOTCONNECT File Upload With S3

To send the high resolution (Preview) image capture by either:

• Pressing Button 1 or Button 2 on the board (black and white micro switches)
• Sending "capture" C2D command to the device.

This will upload the camera snapshot to the /IOTCONNECT AWS S3 bucket.

The uploaded file will appear in the device's Telemetry Files section in the /IOTCONNECT Web UI.

Running WebRTC with /IOTCONNECT and AWS Kinesis Video Streams

The WebRTC streaming support is based on AWS's python-samples-for-amazon-kinesis-video-streams-with-webrtc python kvsWebRTCClientMaster.py example.

To monitor the actual input to the lower resolution (Neural Network) video stream that is being sent to the model for inference, you can use the WebRTC streaming support with /IOTCONNECT.

Simply navigate to the Video Streaming section on the left side menu in the /IOTCONNECT Web UI, and click the Start button.

Note

The choice ot send low resolution stream was made to cover both the preview and the neural network input stream. The app can be modified to send the high resolution preview stream as well.

Pipeline Setup

Quantization can run on either your PC or AWS SageMaker.

It is recommended to use a Python Virtual Environment for this project. The size of installed Python packages can ve quite large, so cleaning up after evaluation should be made simpler.

With the virtual environment created, activate it, and ensure that it is always activated with python libraries ready for the next steps.

Example:

python3 -m venv ~/.demo-venv
source ~/.demo-venv/bin/activate
python3 -m pip install -r pipeline/requirements.txt

When done with the evaluation, simply deactivate the virtual environment and remove the directory.

Image Data Setup

The data/ directory will need to be set up for next steps, so execute the following command:

pipeline/download-images.sh

This process will take a while as the amount of data is quite large. Alternatively, if you do not need to train the dataset and only want to run quantization, you can download the pre-made Calibration Dataset from this link as data/calibration.npz

Running The Quantization Locally

You should install the stedgeai tool so that we can convert your model to NBG (.nb) format. TFLite models can run as well, but will be slightly lower and will take longer to load.

Get familiar with the pipeline application and their options by obtaining help output from the quantize.py script:

cd pipeline/
python3 quantize.py --help
python3 train.py --help

Crate a per-channel and a per-tensor model:

python3 quantize.py # default is per-tensor quantization and file name mobilenetv2-optimized.tflite
python3 quantize.py --output-model=mobilenetv2-optimized-pc.tflite --per-channel
# generate NBG models:
./stedgeai-convert.sh mobilenetv2-optimized
./stedgeai-convert.sh mobilenetv2-optimized-pc

Note

Per-channel quantization is will run slower on the MPx devices but will be slightly more accurate. It is recommended to use the per-tensor quantized models.

Three models will be written in the models/ directory:

• mobilenetv2-optimized.tflite: Per-tensor quantized model.
• mobilenetv2-optimized-pc.tflite: Per-channel quantized model.
• base_model.keras: The input model will be saved for reference.

Additionally, the two TFLite models will be converted to NBG format if you ran the stedgeai-convert.sh script.

Fine-Tuning a Model Training

To obtain the fine-tuned per-tensor TFLite model that is bundled with the application, run the following:

cd pipeline/
python3 train.py # by default, this will save to models/mobilenetv2-finetuned.keras
python3 quantize.py --input-model=mobilenetv2-finetuned.keras --output-model=mobilenetv2-finetuned.tflite
./stedgeai-convert.sh mobilenetv2-finetuned # see below section for more details

When done, upload the new model to the device.

NBG and ONNX Model Support

You should install the stedgeai tool and convert your model to NBG (.nb) format.

Important

The NBG models will start faster and inference significantly faster than any other format. It is recommended to always use this format.

Convert your TFLite models to NBG format with the stedgeai tool:

f=mobilenetv2-optimized
quantization/stedgeai-convert.sh $f
scp models/$f.nb root@$device_ip:app/

It is also possible to convert the TFLite model to ONNX format:

pip install tf2onnx
python -m tf2onnx.convert --opset 16 --tflite models/mobilenetv2-optimized.tflite --output models/mobilenetv2-optimized.onnx

The ONNX model will be expected to run slower than TFLite or an NBG model.

OTA Support

One can pass --send-to=my-device-id as argument to quantize.py script and send the newly converted model to the device directly.

Learn more about how to configure IoTConnect REST API before proceeding at this link.

You can either export the required IoTConnect RESP API account values as environment variables or pass them explicitly to the tool with specific command line arguments that can be obtained when running the tool with --help:

  --iotc-username IOTC_USERNAME
                        Your account username (email). IOTC_USER environment variable can be used instead.
  --iotc-password IOTC_PASSWORD
                        Your account password. IOTC_PASS environment variable can be used instead.
  --iotc-platform IOTC_PLATFORM
                        Account platform ("aws" for AWS, or "az" for Azure). IOTC_PF environment variable can be used instead.
  --iotc-env IOTC_ENV   Account environment - From settings -> Key Vault in the Web UI. IOTC_ENV environment variable can be used instead.
  --iotc-skey IOTC_SKEY
                        Your solution key. IOTC_SKEY environment variable can be used instead.

Example with environment variables set up:

python3 quantize.py --send-to=my-device-id
# or from sagemaker/ directory (see AWS SageMaker setup below):
python3 sagemaker-run.py --send-to=my-device-id

Pipeline on AWS SageMaker

AWS SageMaker can be used for model quantization or fine-tuning the model before quantization with the performance that SageMaker provides.

The sm-quantize.py script provides the integration with AWS CLI to execute quantize.py script within sagemaker, using a similar CLI interface of quantize.py. You would provide the same CLI argument to sagemaker.py that you would to quantize.py.

Similarly, the sm-train.py script provides the fine-tuning support on SageMaker.

AWS Account Setup

AWS CLI will need to be installed and up. Follow this guide to install AWS CLI on your system.

Configure the CLI with:

aws configure

and follow the prompts.

Once the AWS CLI has been set up, set up the environment to run quantization in your account by running commands below. Review the setup-bucket-data.sh script to ensure that it complies with your account admin's policies, or modify it to suit your needs. Alternatively, you can manually set up a sagemaker role for your account and modify the setup-bucket-data.sh accordingly

An example running the full pipeline on SageMaker:

cd sagemaker/
source ~/.demo-venv/bin/activate # form the previously created virtual environment
pip install -r requirements.txt
./setup-bucket-data.sh
python3 sm-train.py # default --output-model=mobilenetv2-finetuned-sm.keras
python3 sm-quantize.py --input-model=mobilenetv2-finetuned-sm.keras --output-model=mobilenetv2-finetuned-sm.tflite

Once each script completes successfully, the generated files will be downloaded to the models/ directory on your local PC.

Note

The setup-bucket-data.sh script will take quite some time to run as it will need to upload about 7 GB of data to your S3 bucket.

Device USB Camera and Troubleshooting

It is possible to use any UVC USB Camera that can provide images at a good frame rate. For example, any Logitech USB camera should suffice. However, tweaks may be needed to get USB Cameras working with the application. The app may need to be modified to select appropriate /dev/video* device and suitable resolution or frame rate.

Ensure to restart the board after plugging in the camera to ensure proper order of camera devices provided by the kernel.

You should verify that your camera appears as /dev/video7. For example:

v4l2-ctl --list-devices
....
HD Pro Webcam C920 (usb-482f0000.usb-1.1):
	/dev/video6
	/dev/video7
	/dev/media2

Otherwise, the application may need to be modified to select the appropriate device.

The application expects the camera devices to be in certain order. If the USB camera happens to be plugged in on boot, this expectation may be invalidated, and we need to re-order the devices. In most cases, unplugging the USB camera nad powering off the device and restarting should solve the problem.

If the application fails to detect the camera, do the following to try reset the state:

• Remove the USB camera.
• Clear the udev data:

rm -rf /run/udev/data/*
rm -rf /var/lib/udev/data/*
poweroff

• Remove the power and plug it back in.