Merge remote-tracking branch 'origin/main' into HEAD

Author: SzabolcsGergely

docs/source/_static/images/api_diagram.png

@@ -3,8 +3,15 @@
 Device
 ======
 
-Device is a DepthAI `module <https://docs.luxonis.com/en/latest/pages/products/>`__. After the :ref:`Pipeline` is defined, it can be uploaded to the device.
-When you create the device in the code, firmware is uploaded together with the pipeline.
+Device represents an `OAK camera <https://docs.luxonis.com/projects/hardware/en/latest/>`__. On all of our devices there's a powerful vision processing unit
+(**VPU**), called `Myriad X <https://www.intel.com/content/www/us/en/products/details/processors/movidius-vpu.html>`__.
+The VPU is optimized for performing AI inference algorithms and for processing sensory inputs (eg. calculating stereo disparity from two cameras).
+
+Device API
+##########
+
+:code:`Device` object represents an OAK device. When starting the device, you have to upload a :ref:`Pipeline` to it, which will get executed on the VPU.
+When you create the device in the code, firmware is uploaded together with the pipeline and other assets (such as NN blobs).
 
 .. code-block:: python
 
@@ -14,8 +21,10 @@ When you create the device in the code, firmware is uploaded together with the p
 
   # Upload the pipeline to the device
   with depthai.Device(pipeline) as device:
-    # Start the pipeline that is now on the device
-    device.startPipeline()
+    # Print Myriad X Id (MxID), USB speed, and available cameras on the device
+    print('MxId:',device.getDeviceInfo().getMxId())
+    print('USB speed:',device.getUsbSpeed())
+    print('Connected cameras:',device.getConnectedCameras())
 
     # Input queue, to send message from the host to the device (you can recieve the message on the device with XLinkIn)
     input_q = device.getInputQueue("input_name", maxSize=4, blocking=False)
@@ -24,7 +33,7 @@ When you create the device in the code, firmware is uploaded together with the p
     output_q = device.getOutputQueue("output_name", maxSize=4, blocking=False)
 
     while True:
-        # Get the message from the queue
+        # Get a message that came from the queue
         output_q.get() # Or output_q.tryGet() for non-blocking
 
         # Send a message to the device
@@ -40,7 +49,7 @@ If you want to use multiple devices on a host, check :ref:`Multiple DepthAI per
 Device queues
 #############
 
-After initializing the device, one has to initialize the input/output queues as well.
+After initializing the device, one has to initialize the input/output queues as well. These queues will be located on the host computer (in RAM).
 
 .. code-block:: python
 
@@ -62,6 +71,40 @@ flags determine the behavior of the queue in this case. You can set these flags
   queue.setMaxSize(10)
   queue.setBlocking(True)
 
+Specifying arguments for :code:`getOutputQueue` method
+######################################################
+
+When obtaining the output queue (example code below), the :code:`maxSize` and :code:`blocking` arguments should be set depending on how
+the messages are intended to be used, where :code:`name` is the name of the outputting stream.
+
+Since queues are on the host computer, memory (RAM) usually isn't that scarce. But if you are using a small SBC like RPI Zero, where there's only 0.5GB RAM,
+you might need to specify max queue size as well.
+
+.. code-block:: python
+
+  with dai.Device(pipeline) as device:
+    queueLeft = device.getOutputQueue(name="manip_left", maxSize=8, blocking=False)
+
+If only the latest results are relevant and previous do not matter, one can set :code:`maxSize = 1` and :code:`blocking = False`.
+That way only latest message will be kept (:code:`maxSize = 1`) and it might also be overwritten in order to avoid waiting for
+the host to process every frame, thus providing only the latest data (:code:`blocking = False`).
+However, if there are a lot of dropped/overwritten frames, because the host isn't able to process them fast enough
+(eg. one-threaded environment which does some heavy computing), the :code:`maxSize` could be set to a higher
+number, which would increase the queue size and reduce the number of dropped frames.
+Specifically, at 30 FPS, a new frame is recieved every ~33ms, so if your host is able to process a frame in that time, the :code:`maxSize`
+could be set to :code:`1`, otherwise to :code:`2` for processing times up to 66ms and so on.
+
+If, however, there is a need to have some intervals of wait between retrieving messages, one could specify that differently.
+An example would be checking the results of :code:`DetectionNetwork` for the last 1 second based on some other event,
+in which case one could set :code:`maxSize = 30` and :code:`blocking = False`
+(assuming :code:`DetectionNetwork` produces messages at ~30FPS).
+
+The :code:`blocking = True` option is mostly used when correct order of messages is needed.
+Two examples would be:
+
+- matching passthrough frames and their original frames (eg. full 4K frames and smaller preview frames that went into NN),
+- encoding (most prominently H264/H265 as frame drops can lead to artifacts).
+
 Blocking behaviour
 ******************
 

docs/source/components/device.rst

@@ -3,12 +3,54 @@
 Messages
 ========
 
-Messages are sent between linked :ref:`Nodes`. The only way nodes communicate with each other is by sending messages from one to another.
+Messages are sent between linked :ref:`Nodes`. The only way nodes communicate with each other is by sending messages from one to another. On the
+table of contents (left side of the page) **all DepthAI messages are listed** under the :code:`Messages` entry. You can click on them to find out more.
 
-If we have :code:`Node1` whose output is linked with :code:`Node2`'s input, a **message** is created in the :code:`Node1`,
-sent out of the :code:`Node1`'s output and to the :code:`Node2`'s input.
+.. rubric:: Creating a message in Script node
 
-On the table of contents (left side of the page) all messages are listed under the :code:`Messages` entry. You can click on them to find out more.
+A DepthAI message can be created either on the device, by a node automatically or manually inside the :ref:`Script` node. In below example,
+the code is taken from the :ref:`Script camera control` example, where :ref:`CameraControl` is created inside the Script node every second
+and sent to the :ref:`ColorCamera`'s input (:code:`cam.inputControl`).
+
+.. code-block:: python
+
+   script = pipeline.create(dai.node.Script)
+   script.setScript("""
+      # Create a message
+      ctrl = CameraControl()
+      # Configure the message
+      ctrl.setCaptureStill(True)
+      # Send the message from the Script node
+      node.io['out'].send(ctrl)
+   """)
+
+.. rubric:: Creating a message on a Host
+
+It can also be created on a host computer and sent to the device via :ref:`XLinkIn` node. :ref:`RGB Camera Control`, :ref:`Video & MobilenetSSD`
+and :ref:`Stereo Depth from host` code examples demonstrate this functionality perfectly. In the example below, we have removed all the code
+that isn't relevant to showcase how a message can be created on the host and sent to the device via XLink.
+
+.. code-block:: python
+
+   # Create XLinkIn node and configure it
+   xin = pipeline.create(dai.node.XLinkIn)
+   xin.setStreamName("frameIn")
+   xin.out.link(nn.input) # Connect it to NeuralNetwork's input
+
+   with dai.Device(pipeline) as device:
+      # Create input queue, which allows you to send messages to the device
+      qIn = device.getInputQueue("frameIn")
+      # Create ImgFrame message
+      img = dai.ImgFrame()
+      img.setData(frame)
+      img.setWidth(300)
+      img.setHeight(300)
+      qIn.send(img) # Send the message to the device
+
+.. rubric:: Creating a message on an external MCU
+
+A message can also be created on an external MCU and sent to the device via :ref:`SPIIn` node. An demo of such functionality is the
+`spi_in_landmark <https://github.com/luxonis/esp32-spi-message-demo/tree/main/spi_in_landmark>`__ example.
 
 .. toctree::
    :maxdepth: 0

docs/source/components/messages.rst

@@ -0,0 +1,32 @@
+StereoDepthConfig
+=================
+
+This message is used to configure the :ref:`StereoDepth` node.
+With this message you can set filters, confidences, thresholds and mode of the :ref:`StereoDepth` node.
+
+Examples of functionality
+#########################
+
+- :ref:`Stereo Depth from host`
+
+Reference
+#########
+
+.. tabs::
+
+  .. tab:: Python
+
+    .. autoclass:: depthai.StereoDepthConfig
+      :members:
+      :inherited-members:
+      :noindex:
+
+  .. tab:: C++
+
+    .. doxygenclass:: dai::StereoDepthConfig
+      :project: depthai-core
+      :members:
+      :private-members:
+      :undoc-members:
+
+.. include::  ../../includes/footer-short.rst

docs/source/components/messages/stereo_depth_config.rst

@@ -3,8 +3,8 @@ Script
 
 Script node allows users to run **custom Python scripts on the device**. Due to the computational resource constraints,
 script node shouldn't be used for heavy computing (eg. image manipulation/CV), but for managing the flow
-of the pipeline. Example use cases would be controlling nodes like :ref:`ImageManip`, :ref:`ColorCamera`, :ref:`SpatialLocationCalculator`,
-decoding :ref:`NeuralNetwork` results, or interfacing with GPIOs.
+of the pipeline (business logic). Example use cases would be controlling nodes like :ref:`ImageManip`, :ref:`ColorCamera`, :ref:`SpatialLocationCalculator`,
+decoding :ref:`NeuralNetwork` results, or interfacing with GPIOs. For **debugging scripts**, we suggest :ref:`Script node logging <script_logging>`.
 
 How to place it
 ###############

docs/source/components/nodes/script.rst

@@ -25,30 +25,50 @@ Inputs and Outputs
 .. code-block::
 
                  ┌───────────────────┐
+                 │                   │ confidenceMap
+                 │                   ├─────────────►
                  │                   │rectifiedLeft
                  │                   ├─────────────►
   left           │                   │   syncedLeft
-  ──────────────►│                   ├─────────────►
+  ──────────────►│-------------------├─────────────►
                  │                   │        depth
                  │                   ├─────────────►
                  │    StereoDepth    │    disparity
                  │                   ├─────────────►
-  right          │                   │rectifiedRight
-  ──────────────►│                   ├─────────────►
-                 │                   │   syncedRight
+  right          │                   │   syncedRight
+  ──────────────►│-------------------├─────────────►
+                 │                   │rectifiedRight
                  │                   ├─────────────►
+  inputConfig    │                   |     outConfig
+  ──────────────►│-------------------├─────────────►
                  └───────────────────┘
 
-**Message types**
+.. tabs::
+
+  .. tab:: **Inputs**
+
+    - :code:`left` - :ref:`ImgFrame` from the left :ref:`MonoCamera`
+    - :code:`right` - :ref:`ImgFrame` from the right :ref:`MonoCamera`
+    - :code:`inputConfig` - :ref:`StereoDepthConfig`
+
+  .. tab:: **Outputs**
+
+    - :code:`confidenceMap` - :ref:`ImgFrame`
+    - :code:`rectifiedLeft` - :ref:`ImgFrame`
+    - :code:`syncedLeft` - :ref:`ImgFrame`
+    - :code:`depth` - :ref:`ImgFrame`
+    - :code:`disparity` - :ref:`ImgFrame`
+    - :code:`rectifiedRight` - :ref:`ImgFrame`
+    - :code:`syncedRight` - :ref:`ImgFrame`
+    - :code:`outConfig` - :ref:`StereoDepthConfig`
 
-- :code:`left` - :ref:`ImgFrame` from the left :ref:`MonoCamera`
-- :code:`right` - :ref:`ImgFrame` from the right :ref:`MonoCamera`
-- :code:`rectifiedLeft` - :ref:`ImgFrame`
-- :code:`syncedLeft` - :ref:`ImgFrame`
-- :code:`depth` - :ref:`ImgFrame`
-- :code:`disparity` - :ref:`ImgFrame`
-- :code:`rectifiedRight` - :ref:`ImgFrame`
-- :code:`syncedRight` - :ref:`ImgFrame`
+  .. tab:: **Debug outputs**
+
+    - :code:`debugDispLrCheckIt1` - :ref:`ImgFrame`
+    - :code:`debugDispLrCheckIt2` - :ref:`ImgFrame`
+    - :code:`debugExtDispLrCheckIt1` - :ref:`ImgFrame`
+    - :code:`debugExtDispLrCheckIt2` - :ref:`ImgFrame`
+    - :code:`debugDispCostDump` - :ref:`ImgFrame`
 
 Internal block diagram of StereoDepth node
 ##########################################
@@ -73,32 +93,36 @@ Currently configurable blocks
 
       .. tab:: Left-Right Check
 
-        Left-Right Check or LR-Check is used to remove incorrectly calculated disparity pixels due to occlusions at object borders (Left and Right camera views
+        **Left-Right Check** or LR-Check is used to remove incorrectly calculated disparity pixels due to occlusions at object borders (Left and Right camera views
         are slightly different).
 
         #. Computes disparity by matching in R->L direction
         #. Computes disparity by matching in L->R direction
         #. Combines results from 1 and 2, running on Shave: each pixel d = disparity_LR(x,y) is compared with disparity_RL(x-d,y). If the difference is above a threshold, the pixel at (x,y) in the final disparity map is invalidated.
 
+        You can use :code:`debugDispLrCheckIt1` and :code:`debugDispLrCheckIt2` debug outputs for debugging/fine-tuning purposes.
+
       .. tab:: Extended Disparity
 
-        The :code:`extended disparity` allows detecting closer distance objects for the given baseline. This increases the maximum disparity search from 96 to 191, meaning the range is now: **[0..190]**.
+        **Extended disparity mode** allows detecting closer distance objects for the given baseline. This increases the maximum disparity search from 96 to 191, meaning the range is now: **[0..190]**.
         So this cuts the minimum perceivable distance in half, given that the minimum distance is now :code:`focal_length * base_line_dist / 190` instead
         of :code:`focal_length * base_line_dist / 95`.
 
         #. Computes disparity on the original size images (e.g. 1280x720)
         #. Computes disparity on 2x downscaled images (e.g. 640x360)
         #. Combines the two level disparities on Shave, effectively covering a total disparity range of 191 pixels (in relation to the original resolution).
 
+        You can use :code:`debugExtDispLrCheckIt1` and :code:`debugExtDispLrCheckIt2` debug outputs for debugging/fine-tuning purposes.
+
       .. tab:: Subpixel Disparity
 
-        Subpixel improves the precision and is especially useful for long range measurements. It also helps for better estimating surface normals.
+        **Subpixel mode** improves the precision and is especially useful for long range measurements. It also helps for better estimating surface normals.
 
         Besides the integer disparity output, the Stereo engine is programmed to dump to memory the cost volume, that is 96 levels (disparities) per pixel,
-        then software interpolation is done on Shave, resulting a final disparity with 5 fractional bits, resulting in significantly more granular depth
-        steps (32 additional steps between the integer-pixel depth steps), and also theoretically, longer-distance depth viewing - as the maximum depth
-        is no longer limited by a feature being a full integer pixel-step apart, but rather 1/32 of a pixel. In this mode, stereo cameras perform: :code:`96 depth steps * 32 subpixel depth steps = 3,072 depth steps.`
-        Note that Subpixel and Extended Disparity are not yet supported simultaneously (which would result in :code:`191 * 32 = 6,112 depth steps`), but should be available in the near future (`Pull Request <https://github.com/luxonis/depthai-python/pull/347>`__).
+        then software interpolation is done on Shave, resulting a final disparity with 3 fractional bits, resulting in significantly more granular depth
+        steps (8 additional steps between the integer-pixel depth steps), and also theoretically, longer-distance depth viewing - as the maximum depth
+        is no longer limited by a feature being a full integer pixel-step apart, but rather 1/8 of a pixel. In this mode, stereo cameras perform: :code:`94 depth steps * 8 subpixel depth steps + 2 (min/max values) = 754 depth steps`
+        Note that Subpixel and Extended Disparity are not yet supported simultaneously.
 
         For comparison of normal disparity vs. subpixel disparity images, click `here <https://github.com/luxonis/depthai/issues/184>`__.
 
@@ -138,12 +162,7 @@ Currently configurable blocks
 Current limitations
 ###################
 
-If one or more of the additional depth modes (:code:`lrcheck`, :code:`extended`, :code:`subpixel`) are enabled, then:
-
-- median filtering is disabled on device
-- with subpixel, if both :code:`depth` and :code:`disparity` are used, only :code:`depth` will have valid output
-
-Otherwise, :code:`depth` output is **U16** (in millimeters) and median is functional.
+- Median filtering is disabled when subpixel mode is set to 4 or 5 bits.
 
 Stereo depth FPS
 ################