Image preprocessing on Sense Craft AI

[yousefbilal]

I have trained a MobileNetV2 model on a custom dataset following the example configurations. Testing the quantized model on the dataset gives good results reaching above 90% in accuracy. However, when I deploy the model using Sense Craft AI on Grove Vision AI V2, I get a 99% probability on one of the classes all the time, except when I black out the camera it drops down to ~83%. I uploaded the same model tflite model to Edge Impulse and tested the model on few images of the dataset and it appeared to work correctly when the input was set to values 0..1 . Is it possible that the model is performing poorly on camera images because of a wrong preprocessing withing the Grove firmware?

[iChizer0]

Hi @yousefbilal at present, all the pre-processing on the firmware side is consistent, and if this is the case, other models may also appear abnormal, yet we didn't observe. If it's convenient, please send us the TFlite model and Vela model you are testing, and we will validate these models and try to solve your issue, thanks.

[yousefbilal]

I think I have solved the issue for now, but can you let me know what preprocessing is happening on the firmware? Also, if possible, can you point me to the source code of the firmware that Sense Craft flashes into the device?

[iChizer0]

@yousefbilal The source code repository for the firmware is located at https://github.com/Seeed-Studio/sscma-example-we2/tree/main/EPII_CM55M_APP_S/library, which name contains sscma as our component.

The current preprocessing is divided into two steps, first converting the image of yuv422 to rgb888, which includes an automatic resize operation (inter_nearest may introduce some errors for computational considerations), and then mapping rgb888 to a quantized space (for int8 symmetrically quantization we simply subtract 128), some of the code can be referenced here:
https://github.com/Seeed-Studio/SSCMA-Micro/blob/415035c943620de9d3380215af81a1300215b014/sscma/core/model/ma_model_classifier.cpp#L87 .

[yousefbilal]

Thanks for the info. Are the vela models similarly built to match the quantized input?

[iChizer0]

@yousefbilal Yes, the input and output of the Vela model must be at least consistent with the quantization of the exported TFLite model (int8 symmetrically quantized), and the Vela converter generally does not automatically modify the input-related parameters. Strictly speaking, the implementation standard of Vela is consistent with TFLM (the supported OP features are a subset of TFL), so if you want to get the correct calculation results, the quantized TFLite model in TFLM and TFL inference results need to be as same as possible.