Merge pull request #302 from luxonis/develop

Release 2.6.0.0

Author: SzabolcsGergely

depthai-core

@@ -62,7 +62,7 @@ as:
 (this confidence score is kind-of inverted, if say comparing with NN)
 
 For the final disparity map, a filtering is applied based on the confidence threshold value: the pixels that have their confidence score larger than
-the threshold get invalidated, i.e. their disparity value is set to zero. You can set the confidence threshold with :code:`stereo.setConfidenceThreshold()`.
+the threshold get invalidated, i.e. their disparity value is set to zero. You can set the confidence threshold with :code:`stereo.initialConfig.setConfidenceThreshold()`.
 
 Current limitations
 ###################

docs/source/components/nodes/stereo_depth.rst

@@ -0,0 +1,38 @@
+Edge detector
+=============
+
+This example performs edge detection on 3 different inputs: left, right and RGB camera.
+HW accelerated sobel filter 3x3 is used.
+Sobel filter parameters can be changed by keys 1 and 2.
+
+Demo
+####
+
+
+Setup
+#####
+
+.. include::  /includes/install_from_pypi.rst
+
+Source code
+###########
+
+.. tabs::
+
+    .. tab:: Python
+
+        Also `available on GitHub <https://github.com/luxonis/depthai-python/blob/main/examples/edge_detector.py>`__
+
+        .. literalinclude:: ../../../examples/edge_detector.py
+           :language: python
+           :linenos:
+
+    .. tab:: C++
+
+        Also `available on GitHub <https://github.com/luxonis/depthai-core/blob/main/examples/src/edge_detector.cpp>`__
+
+        .. literalinclude:: ../../../depthai-core/examples/src/edge_detector.cpp
+           :language: cpp
+           :linenos:
+
+.. include::  /includes/footer-short.rst

docs/source/samples/edge_detector.rst

@@ -29,6 +29,7 @@ Code samples are used for automated testing. They are also a great starting poin
 - :ref:`Video & MobilenetSSD` - Runs MobileNetSSD on the video from the host
 - :ref:`IMU Accelerometer & Gyroscope` - Accelerometer and gyroscope at 500hz rate
 - :ref:`IMU Rotation Vector` - Rotation vector at 400 hz rate
+- :ref:`Edge detector` - Edge detection on input frame
 
 .. rubric:: Complex
 

docs/source/tutorials/code_samples.rst

@@ -70,9 +70,10 @@ Let's verify we're able to load all of our dependencies. Open the :code:`hello_w
 
 .. code-block:: python
 
-  import numpy as np # numpy - manipulate the packet data returned by depthai
-  import cv2 # opencv - display the video stream
-  import depthai # access the camera and its data packets
+  import numpy as np  # numpy - manipulate the packet data returned by depthai
+  import cv2  # opencv - display the video stream
+  import depthai  # depthai - access the camera and its data packets
+  import blobconverter  # blobconverter - compile and download MyriadX neural network blobs
 
 Try running the script and ensure it executes without error:
 
@@ -111,13 +112,16 @@ Now, first node we will add is a :class:`ColorCamera`. We will use the :code:`pr
   cam_rgb.setPreviewSize(300, 300)
   cam_rgb.setInterleaved(False)
 
-Up next, let's define a :class:`NeuralNetwork` node with `mobilenet-ssd network <https://docs.openvinotoolkit.org/latest/omz_models_public_mobilenet_ssd_moblenet_ssd.html>`__.
-The blob file for this example can be found `here <https://github.com/luxonis/depthai-tutorials/raw/e37989e07a36a57ffef624b7aa8cf20ab493fa07/1-hello-world/mobilenet-ssd/mobilenet-ssd.blob>`__
+Up next, let's define a :class:`MobileNetDetectionNetwork` node with `mobilenet-ssd network <https://docs.openvinotoolkit.org/latest/omz_models_public_mobilenet_ssd_moblenet_ssd.html>`__.
+The blob file for this example will be compiled automatically using `blobconverter tool <https://pypi.org/project/blobconverter/>`__, we'll be provided with a ready-to-use blob path.
+With this node, the output from nn will be parsed on device side and we'll receive a ready to use detection objects. For this to work properly, we need also to set the confidence threshold
+to filter out the incorrect results
 
 .. code-block:: python
 
-  detection_nn = pipeline.createNeuralNetwork()
-  detection_nn.setBlobPath("/path/to/mobilenet-ssd.blob")
+  detection_nn = pipeline.createMobileNetDetectionNetwork()
+  detection_nn.setBlobPath(str(blobconverter.from_zoo(name='mobilenet-ssd', shaves=6)))
+  detection_nn.setConfidenceThreshold(0.5)
 
 And now, let's connect a color camera :code:`preview` output to neural network input
 
@@ -143,12 +147,11 @@ and in our case, since we want to receive data from device to host, we will use
 Initialize the DepthAI Device
 #############################
 
-Having the pipeline defined, we can now initialize a device and start it
+Having the pipeline defined, we can now initialize a device with pipeline and start it
 
 .. code-block:: python
 
-  device = depthai.Device(pipeline)
-  device.startPipeline()
+  with depthai.Device(pipeline) as device:
 
 .. note::
 
@@ -158,7 +161,7 @@ Having the pipeline defined, we can now initialize a device and start it
 
   .. code-block:: python
 
-    device = depthai.Device(pipeline, True)
+    device = depthai.Device(pipeline, usb2mode=True)
 
 
 
@@ -181,19 +184,22 @@ for rgb frame and one for nn results
 .. code-block:: python
 
   frame = None
-  bboxes = []
+  detections = []
 
 Also, due to neural network implementation details, bounding box coordinates in inference results are represented
 as floats from <0..1> range - so relative to frame width/height (e.g. if image has 200px width and nn returned x_min
 coordinate equal to 0.2, this means the actual (normalised) x_min coordinate is 40px).
 
-That's why we need to define a helper function, :code:`frame_form`, that will convert these <0..1> values into actual
+That's why we need to define a helper function, :code:`frameNorm`, that will convert these <0..1> values into actual
 pixel positions
 
 .. code-block:: python
 
-  def frame_norm(frame, bbox):
-      return (np.array(bbox) * np.array([*frame.shape[:2], *frame.shape[:2]])[::-1]).astype(int)
+    def frameNorm(frame, bbox):
+        normVals = np.full(len(bbox), frame.shape[0])
+        normVals[::2] = frame.shape[1]
+        return (np.clip(np.array(bbox), 0, 1) * normVals).astype(int)
+
 
 Consuming the results
 #####################
@@ -215,65 +221,22 @@ Now, inside this loop, first thing to do is fetching latest results from both nn
 The :code:`tryGet` method returns either the latest result or :code:`None` if the queue is empty.
 
 Results, both from rgb camera or neural network, will be delivered as 1D arrays, so both of them will require transformations
-to be useful for display (we have already defined one of the transformations needed - the :code:`frame_norm` function)
+to be useful for display (we have already defined one of the transformations needed - the :code:`frameNorm` function)
 
-First up, if we receive a frame from rgb camera, we need to convert it from 1D array into HWC form (HWC stands for
-Height Width Channels, so 3D array, with first dimension being width, second height, and third the color channel)
+First up, if we receive a frame from rgb camera using the :code:`getCvFrame` command
 
 .. code-block:: python
 
   if in_rgb is not None:
-      shape = (3, in_rgb.getHeight(), in_rgb.getWidth())
-      frame = in_rgb.getData().reshape(shape).transpose(1, 2, 0).astype(np.uint8)
-      frame = np.ascontiguousarray(frame)
+      frame = in_rgb.getCvFrame()
 
-Second, the neural network results will also need transformations. These are also returned as a 1D array, but this time
-the array has a fixed size (constant, no matter how many results the neural network has actually produced).
-Actual results in array are followed with :code:`-1` and then filled to meet the fixed size with :code:`0`.
-One results has 7 fields, each being respectively :code:`image_id, label, confidence, x_min, y_min, x_max, y_max`.
-We will want only the last four values (being the bounding box), but we'll also filter out the ones which :code:`confidence`
-is below a certain threshold - it can be anywhere between <0..1>, and for this example we will use :code:`0.8` threshold
+Second, we will receive the neural network results. Default MobileNetSSD result has 7 fields, each being respectively :code:`image_id, label, confidence, x_min, y_min, x_max, y_max`,
+and by accessing the :code:`detections` array, we receive the detection objects that allow us to access these fields
 
 .. code-block:: python
 
   if in_nn is not None:
-      bboxes = np.array(in_nn.getFirstLayerFp16())
-      bboxes = bboxes[:np.where(bboxes == -1)[0][0]]
-      bboxes = bboxes.reshape((bboxes.size // 7, 7))
-      bboxes = bboxes[bboxes[:, 2] > 0.8][:, 3:7]
-
-To better understand this flow, let's take an example. Let's assume the :code:`np.array(in_nn.getFirstLayerFp16())` returns the following array
-
-.. code-block:: python
-
-  [0, 15, 0.99023438, 0.45556641, 0.34399414  0.88037109, 0.9921875, 0, 15, 0.98828125, 0.03076172, 0.23388672, 0.60205078, 1.0078125, -1, 0, 0, 0, ...]
-
-First operation, :code:`bboxes[:np.where(bboxes == -1)[0][0]]`, removes the trailing zeros from the array, so now the bbox array will look like this
-
-.. code-block:: python
-
-  [0, 15, 0.99023438, 0.45556641, 0.34399414  0.88037109, 0.9921875, 0, 15, 0.98828125, 0.03076172, 0.23388672, 0.60205078, 1.0078125]
-
-Second one - :code:`bboxes.reshape((bboxes.size // 7, 7))`, reshapes the 1D array into 2D array - where each row is a separate result
-
-.. code-block:: python
-
-  [
-    [0, 15, 0.99023438, 0.45556641, 0.34399414  0.88037109, 0.9921875],
-    [0, 15, 0.98828125, 0.03076172, 0.23388672, 0.60205078, 1.0078125]
-  ]
-
-Last one - :code:`bboxes = bboxes[bboxes[:, 2] > 0.8][:, 3:7]` - will filter the results based on the confidence column (3rd one, with index :code:`2`)
-to be above a defined threshold (:code:`0.8`) - and from these results, it will only take the last 4 columns being the bounding boxes.
-Since both our results have a very high confidence (:code:`0.99023438` and :code:`0.98828125` respectively), they won't be filtered, and the final
-array will look like this
-
-.. code-block:: python
-
-  [
-    [0.45556641, 0.34399414  0.88037109, 0.9921875],
-    [0.03076172, 0.23388672, 0.60205078, 1.0078125]
-  ]
+      detections = in_nn.detections
 
 Display the results
 ###################
@@ -283,12 +246,12 @@ Up to this point, we have all our results consumed from the DepthaI device, and
 .. code-block:: python
 
   if frame is not None:
-      for raw_bbox in bboxes:
-          bbox = frame_norm(frame, raw_bbox)
+      for detection in detections:
+          bbox = frameNorm(frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))
           cv2.rectangle(frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), (255, 0, 0), 2)
       cv2.imshow("preview", frame)
 
-You can see here the usage of :code:`frame_norm` we defined earlier for bounding box coordinates normalization.
+You can see here the usage of :code:`frameNorm` we defined earlier for bounding box coordinates normalization.
 By using :code:`cv2.rectangle` we draw a rectangle on the rgb frame as an indicator where the face position is, and then
 we display the frame using :code:`cv2.imshow`
 

docs/source/tutorials/hello_world.rst

@@ -275,8 +275,8 @@ In particular, let's change the :code:`setBlobPath` invocation to load our model
 
 .. code-block:: diff
 
-  - detection_nn.setBlobPath("/path/to/mobilenet-ssd.blob")
-  - detection_nn.setBlobPath("/path/to/face-detection-retail-0004.blob")
+  - detection_nn.setBlobPath(str(blobconverter.from_zoo(name='mobilenet-ssd', shaves=6)))
+  + detection_nn.setBlobPath("/path/to/face-detection-retail-0004.blob")
 
 And that's all!
 

docs/source/tutorials/local_convert_openvino.rst

@@ -21,7 +21,7 @@ Simple
    ../samples/video_mobilenet.rst
    ../samples/imu_accelerometer_gyroscope.rst
    ../samples/imu_rotation_vector.rst
-
+   ../samples/edge_detector.rst
 
 These samples are great starting point for the gen2 API.
 
@@ -38,3 +38,4 @@ These samples are great starting point for the gen2 API.
 - :ref:`RGB & MobileNetSSD @ 4K` - Runs MobileNetSSD on RGB frames and displays detections on both preview and 4k frames
 - :ref:`Mono & MobilenetSSD` - Runs MobileNetSSD on mono frames and displays detections on the frame
 - :ref:`Video & MobilenetSSD` - Runs MobileNetSSD on the video from the host
+- :ref:`Edge detector` - Edge detection on input frame

docs/source/tutorials/simple_samples.rst

@@ -124,7 +124,5 @@ add_python_example(stereo_depth_from_host stereo_depth_from_host.py)
 add_python_example(stereo_depth_video stereo_depth_video.py)
 add_python_example(imu_gyroscope_accelerometer imu_gyroscope_accelerometer.py)
 add_python_example(imu_rotation_vector imu_rotation_vector.py)
-add_python_example(calibration_flash_v5 calibration_flash_v5.py)
-add_python_example(calibration_flash calibration_flash.py)
-add_python_example(calibration_load calibration_load.py)
-add_python_example(calibration_reader calibration_reader.py)
+add_python_example(rgb_depth_aligned rgb_depth_aligned.py)
+add_python_example(edge_detector edge_detector.py)

examples/CMakeLists.txt

@@ -39,7 +39,7 @@
 
 manip.initialConfig.setCropRect(topLeft.x, topLeft.y, bottomRight.x, bottomRight.y)
 manip.setMaxOutputFrameSize(monoRight.getResolutionHeight()*monoRight.getResolutionWidth()*3)
-stereo.setConfidenceThreshold(200)
+stereo.initialConfig.setConfidenceThreshold(200)
 
 # Linking
 configIn.out.link(manip.inputConfig)

examples/depth_crop_control.py

@@ -29,9 +29,9 @@
 monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT)
 
 # Create a node that will produce the depth map (using disparity output as it's easier to visualize depth this way)
-depth.setConfidenceThreshold(200)
+depth.initialConfig.setConfidenceThreshold(200)
 # Options: MEDIAN_OFF, KERNEL_3x3, KERNEL_5x5, KERNEL_7x7 (default)
-depth.setMedianFilter(dai.StereoDepthProperties.MedianFilter.KERNEL_7x7)
+depth.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_7x7)
 depth.setLeftRightCheck(lr_check)
 depth.setExtendedDisparity(extended_disparity)
 depth.setSubpixel(subpixel)