You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: documentation/asciidoc/accessories/ai-camera/model-conversion.adoc
+29-27Lines changed: 29 additions & 27 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,46 +2,35 @@
2
2
3
3
To deploy a new neural network model to the Raspberry Pi AI Camera, complete the following steps:
4
4
5
-
. Provide a neural network model.
6
-
. Quantise and compress the model so that it can run using the resources available on the IMX500 camera module.
7
-
. Convert the compressed model to IMX500 format.
5
+
. Provide a floating-point neural network model (PyTorch or TensorFlow).
6
+
. Run the model through Edge-MDT (Edge AI Model Development Toolkit).
7
+
.. *Quantise* and compress the model so that it can run using the resources available on the IMX500 camera module.
8
+
.. *Convert* the compressed model to IMX500 format.
8
9
. Package the model into a firmware file that can be loaded at runtime onto the camera.
9
10
10
-
The first three steps will normally be performed on a more powerful computer such as a desktop or server. You must run the final packaging step on a Raspberry Pi.
11
+
The first two steps will normally be performed on a more powerful computer such as a desktop or server. You must run the final packaging step on a Raspberry Pi.
11
12
12
13
=== Model creation
13
14
14
-
The creation of neural network models is beyond the scope of this guide. Existing models can be re-used, or new ones created using popular frameworks like TensorFlow or PyTorch.
15
+
The creation of neural network models is beyond the scope of this guide. Existing models can be re-used, or new ones created using popular AI frameworks like TensorFlow or PyTorch.
15
16
16
17
For more information, see the official https://developer.aitrios.sony-semicon.com/en/raspberrypi-ai-camera[AITRIOS developer website].
17
18
18
-
=== Quantisation and compression
19
+
=== Model compression and conversion
19
20
20
-
Models are quantised and compressed using Sony's Model Compression Toolkit. To install the toolkit, run the following command:
21
+
==== Edge-MDT installation
21
22
22
-
[source,console]
23
-
----
24
-
$ pip install model_compression_toolkit
25
-
----
26
-
27
-
For more information, see the https://github.com/sony/model_optimization[Sony model optimization GitHub repository].
28
-
29
-
The Model Compression Toolkit generates a quantised model in the following formats:
23
+
The Edge-MDT (Model Development Toolkit) software package installs all the tools required to quantise, compress, and convert models to run on your IMX500 device.
30
24
31
-
* Keras (TensorFlow)
32
-
* ONNX (PyTorch)
33
-
34
-
=== Conversion
35
-
36
-
To convert a model, first install the converter tools:
25
+
The Edge-MDT package takes a parameter to select between installing the PyTorch or TensorFlow version of the tools.
37
26
38
27
[tabs]
39
28
======
40
29
TensorFlow::
41
30
+
42
31
[source,console]
43
32
----
44
-
$ pip install imx500-converter[tf]
33
+
$ pip install edge-mdt[tf]
45
34
----
46
35
+
47
36
TIP: Always use the same version of TensorFlow you used to compress your model.
@@ -50,13 +39,26 @@ PyTorch::
50
39
+
51
40
[source,console]
52
41
----
53
-
$ pip install imx500-converter[pt]
42
+
$ pip install edge-mdt[pt]
54
43
----
55
44
======
56
45
57
-
If you need to install both packages, use two separate Python virtual environments. This prevents TensorFlow and PyTorch from causing conflicts with one another.
46
+
If you need to install both packages, use two separate Python virtual environments. This prevents TensorFlow and PyTorch from causing conflicts with each other.
47
+
48
+
==== Model Optimization
58
49
59
-
Next, convert the model:
50
+
Models are quantised and compressed using Sony's Model Compression Toolkit (MCT). This tool is automatically installed as part of the Edge-MDT installation step. For more information, see the https://github.com/sony/model_optimization[Sony model optimization GitHub repository].
51
+
52
+
The Model Compression Toolkit generates a quantised model in the following formats:
53
+
54
+
* Keras (TensorFlow)
55
+
* ONNX (PyTorch)
56
+
57
+
=== Conversion
58
+
59
+
The converter is a command line application that compiles the quantised model (in .onnx or .keras formats) into a binary file that can be packaged and loaded onto the AI Camera. This tool is automatically installed as part of the Edge-MDT installation step.
Both commands create an output folder that contains a memory usage report and a `packerOut.zip` file.
80
+
IMPORTANT: For optimal use of the memory available to the accelerator on the IMX500 sensor, add `--no-input-persistency` to the above commands. However, this will disable input tensor generation that may be used for debugging purposes.
79
81
80
-
For optimal use of the memory available to the accelerator on the IMX500 sensor, add `--no-input-persistency` to the above commands. However, this will disable input tensor generation and return to the application for debugging purposes.
82
+
Both commands create an output folder that contains a memory usage report and a `packerOut.zip` file.
81
83
82
84
For more information on the model conversion process, see the official https://developer.aitrios.sony-semicon.com/en/raspberrypi-ai-camera/documentation/imx500-converter[Sony IMX500 Converter documentation].
0 commit comments