Skip to content

Commit c08575b

Browse files
nathan-continonaushir
authored andcommitted
Copy edit initial ai camera documentation
1 parent cebfc6b commit c08575b

File tree

10 files changed

+106
-74
lines changed

10 files changed

+106
-74
lines changed

documentation/asciidoc/accessories/ai-camera/about.adoc

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
[[ai-camera]]
22
== About
33

4-
The Raspberry Pi AI Camera using the Sony IMX500 imaging sensor is a great way of adding low-latency and high-performance AI capabilities to any camera application. The tight integration with https://www.raspberrypi.com/documentation/computers/camera_software.html[Raspberry Pi's camera software stack] allows users to deploy their own neural network models with minimal effort.
4+
The Raspberry Pi AI Camera uses the Sony IMX500 imaging sensor to provide low-latency and high-performance AI capabilities to any camera application. The tight integration with https://www.raspberrypi.com/documentation/computers/camera_software.adoc[Raspberry Pi's camera software stack] allows users to deploy their own neural network models with minimal effort.
5+
6+
image::images/ai-camera.png[The Raspberry Pi AI Camera]
7+
8+
This section demonstrates how to run either a pre-packaged or custom neural network model on the camera. Additionally, this section includes the steps required to interpret inference data generated by neural networks running on the IMX500 in https://github.com/raspberrypi/rpicam-apps[`rpicam-apps`] and https://github.com/raspberrypi/picamera2[Picamera2].
59

6-
This tutorial goes through the steps necessary for running either a pre-packaged or custom made neural network model on the camera. It also goes through the steps required for interperting the inference data generated by neural networks running on the IMX500 in https://github.com/raspberrypi/rpicam-apps[`rpicam-apps`] and https://github.com/raspberrypi/picamera2[`Picamera2`].

documentation/asciidoc/accessories/ai-camera/details.adoc

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,17 @@ The diagram below shows the various camera software components (in green) used d
55

66
image::images/imx500-block-diagram.svg[IMX500 block diagram]
77

8-
On startup, the IMX500 sensor modules gets loaded with firmware to run a particular neural network model. On streaming, the IMX500 generates an image stream together with an inference stream. This inference stream holds the inputs and outputs of the neural network model, also known as input/output tensors.
8+
At startup, the IMX500 sensor module loads firmware to run a particular neural network model. During streaming, the IMX500 generates _both_ an image stream and an inference stream. This inference stream holds the inputs and outputs of the neural network model, also known as input/output **tensors**.
99

1010
=== Device drivers
1111

12-
At the lowest level, the camera module is configured over the I2C bus from the IMX500 sensor kernel driver. The CSI2 driver (`CFE` on Pi 5, `Unicam` on all other Pi platforms) sets up the receiver to write the image data stream into a frame buffer, together with the embedded data and inference data streams into another buffer in memory.
12+
At the lowest level, the the IMX500 sensor kernel driver configures the camera module over the I2C bus. The CSI2 driver (`CFE` on Pi 5, `Unicam` on all other Pi platforms) sets up the receiver to write the image data stream into a frame buffer, together with the embedded data and inference data streams into another buffer in memory.
1313

14-
The firmware files are also transferred over the I2C bus wires with either the standard I2C protocol, or a custom high speed protocol in the case of Pi 5. This is handled through the RP2040 SPI driver in the kernel. The RP2040 microcontroller is responsible for management of the firmware transfer operation on the camera module. It bridges the I2C transfers from the kernel to the IMX500 via a SPI bus. The RP2040 also caches firmware files on the on-board flash chip for fast upload to the IMX500, avoiding the need to transfer the entire firmware blob over the I2C bus.
14+
The firmware files also transfer over the I2C bus wires. On most devices, this uses the standard I2C protocol, but Raspberry Pi 5 uses a custom high speed protocol. The RP2040 SPI driver in the kernel handles firmware file transfer, since the transfer uses the RP2040 microcontroller. The microcontroller bridges the I2C transfers from the kernel to the IMX500 via a SPI bus. Additionally, the RP2040 caches firmware files in on-board storage. This avoids the need to transfer entire firmware blobs over the I2C bus, significantly speeding up firmware loading for firmware you've already used.
1515

16-
=== libcamera
16+
=== `libcamera`
1717

18-
Once `libcamera` dequeues the image and inference data buffers from the kernel, the IMX500 specific `cam-helper` library (part of the Raspberry Pi IPA within libcamera), parses the inference buffer to get the input/output tensors. These tensors as packaged as Raspberry Pi vendor specific https://libcamera.org/api-html/namespacelibcamera_1_1controls.html[`libcamera Controls`] to be returned out to the application for consumption. The following controls are returned:
18+
Once `libcamera` dequeues the image and inference data buffers from the kernel, the IMX500 specific `cam-helper` library (part of the Raspberry Pi IPA within `libcamera`) parses the inference buffer to access the input/output tensors. These tensors are packaged as Raspberry Pi vendor-specific https://libcamera.org/api-html/namespacelibcamera_1_1controls.html[`libcamera` controls]. `libcamera` returns the following controls:
1919

2020
[%header,cols="a,a"]
2121
|===
@@ -61,9 +61,9 @@ struct CnnInputTensorInfo {
6161

6262
|===
6363

64-
=== rpicam-apps
64+
=== `rpicam-apps`
6565

66-
`rpicam-apps` provides an IMX500 postprocessing stage base class that implements helpers for IMX500 postprocessing stages - https://github.com/raspberrypi/rpicam-apps/blob/post_processing_stages/imx500_post_processing_stage.hpp[`IMX500PostProcessingStage`]. This base class can be used to derive a new postprocessing stage for any neural network model running on the IMX500. For example, in https://github.com/raspberrypi/rpicam-apps/blob/post_processing_stages/imx500_mobilenet_ssd.cpp[`imx500_mobilenet_ssd.cpp`], the following derived class is instanciated (not complete implementation):
66+
`rpicam-apps` provides an IMX500 post-processing stage base class that implements helpers for IMX500 post-processing stages: https://github.com/raspberrypi/rpicam-apps/blob/post_processing_stages/imx500_post_processing_stage.hpp[`IMX500PostProcessingStage`]. Use this base class to derive a new post-processing stage for any neural network model running on the IMX500. For an example, see https://github.com/raspberrypi/rpicam-apps/blob/post_processing_stages/imx500_mobilenet_ssd.cpp[`imx500_mobilenet_ssd.cpp`]:
6767

6868
[source,cpp]
6969
----
@@ -82,7 +82,7 @@ public:
8282
};
8383
----
8484

85-
On every frame received by the application, the `Process()` function is called (`ObjectInference::Process()` in the above case). In this function, you can extract the output tensor for futher processing and/or analysis by the stage:
85+
For every frame received by the application, the `Process()` function is called (`ObjectInference::Process()` in the above case). In this function, you can extract the output tensor for further processing or analysis:
8686

8787
[source,cpp]
8888
----
@@ -96,23 +96,23 @@ if (!output)
9696
std::vector<float> output_tensor(output->data(), output->data() + output->size());
9797
----
9898

99-
Once this is completed, the final results can either be visualised or saved in metadata and consumed by either another downstream stage, or the top level application itself. In the object inference case:
99+
Once completed, the final results can either be visualised or saved in metadata and consumed by either another downstream stage, or the top level application itself. In the object inference case:
100100

101101
[source,cpp]
102102
----
103103
if (objects.size())
104104
completed_request->post_process_metadata.Set("object_detect.results", objects);
105105
----
106106

107-
The `object_detect_draw_cv` postprocessing stage running downstream fetches these results from the metadata and draws the bounding boxes onto the image in the `ObjectDetectDrawCvStage::Process()` function:
107+
The `object_detect_draw_cv` post-processing stage running downstream fetches these results from the metadata and draws the bounding boxes onto the image in the `ObjectDetectDrawCvStage::Process()` function:
108108

109109
[source,cpp]
110110
----
111111
std::vector<Detection> detections;
112112
completed_request->post_process_metadata.Get("object_detect.results", detections);
113113
----
114114

115-
A full list of helper functions provided by `IMX500PostProcessingStage` is listed below:
115+
The following table contains a full list of helper functions provided by `IMX500PostProcessingStage`:
116116

117117
[%header,cols="a,a"]
118118
|===
@@ -145,7 +145,7 @@ There are a number of scaling/cropping/translation operations occurring from the
145145

146146
=== Picamera2
147147

148-
IMX500 integration in `Picamera2` is very similar to what is available in `rpicam-apps`. `Picamera2` has an IMX500 helper class that provides the same functionality as the `rpicam-apps` `IMX500PostProcessingStage` base class. This can be imported to any python script with:
148+
IMX500 integration in Picamera2 is very similar to what is available in `rpicam-apps`. Picamera2 has an IMX500 helper class that provides the same functionality as the `rpicam-apps` `IMX500PostProcessingStage` base class. This can be imported to any python script with:
149149

150150
[source,python]
151151
----
@@ -155,7 +155,7 @@ from picamera2.devices.imx500 import IMX500
155155
imx500 = IMX500(model_file)
156156
----
157157

158-
To retrieve the output tensors, you fetch them from the controls and use it for for futher processing and/or analysis by the python script.
158+
To retrieve the output tensors, fetch them from the controls. You can then apply additional processing in your python script.
159159

160160
For example, in an object inference use case such as https://github.com/raspberrypi/picamera2/tree/main/examples/imx500/imx500_object_detection_demo.py[imx500_object_detection_demo.py], the object bounding boxes and confidence values are extracted in `parse_detections()` and draw the boxes on the image in `draw_detections()`:
161161

@@ -192,9 +192,9 @@ def parse_detections(request, stream='main'):
192192
draw_detections(request, detections, stream)
193193
----
194194

195-
Note that there is no additional hysteresis or temporal filtering applied to the output like in the `rpicam-apps` example. However this should be easy enough to add to this example if needed.
195+
Unlike the `rpicam-apps` example, this example applies no additional hysteresis or temporal filtering.
196196

197-
The IMX500 class in `Picamera2` provides the following helper functions:
197+
The IMX500 class in Picamera2 provides the following helper functions:
198198

199199
[%header,cols="a,a"]
200200
|===

0 commit comments

Comments
 (0)