Skip to content

Commit b3c798a

Browse files
Erol444Erol444
authored
Merge pull request #489 from luxonis/add_NN_examples
Add nn examples
2 parents 929a9c1 + 600e6a1 commit b3c798a

File tree

9 files changed

+286
-1
lines changed

9 files changed

+286
-1
lines changed

‎depthai-core‎

‎docs/source/_static/images/examples/normalize_model.png‎

13.6 KB
Loading

‎docs/source/samples/NeuralNetwork/concat_multi_input.rst‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
Multi-Input Frame Concationation
2+
================================
3+
4+
Example concatenates all 3 inputs with a simple custom model created with PyTorch (`link here <https://github.com/luxonis/depthai-experiments/blob/master/gen2-custom-models/generate_model/pytorch_concat.py>`__,
5+
`tutorial here <https://docs.luxonis.com/en/latest/pages/tutorials/creating-custom-nn-models/>`__).
6+
It uses :ref:`NeuralNetwork`'s multiple input feature and links all 3 camera streams directly to the NeuralNetwork node.
7+
8+
Demo
9+
####
10+
11+
.. image:: https://user-images.githubusercontent.com/18037362/134209980-09c6e2f9-8a26-45d5-a6ad-c31d9e2816e1.png
12+
13+
Setup
14+
#####
15+
16+
.. include:: /includes/install_from_pypi.rst
17+
18+
Source code
19+
###########
20+
21+
.. tabs::
22+
23+
.. tab:: Python
24+
25+
Also `available on GitHub <https://github.com/luxonis/depthai-python/blob/main/examples/NeuralNetwork/concat_multiple_input.py>`__
26+
27+
.. literalinclude:: ../../../../examples/NeuralNetwork/concat_multi_input.py
28+
:language: python
29+
:linenos:
30+
31+
.. tab:: C++
32+
33+
Also `available on GitHub <https://github.com/luxonis/depthai-core/blob/main/examples/src/concat_multiple_input.cpp>`__
34+
35+
.. literalinclude:: ../../../../depthai-core/examples/NeuralNetwork/concat_multi_input.cpp
36+
:language: cpp
37+
:linenos:
38+
39+
.. include:: /includes/footer-short.rst

‎docs/source/samples/NeuralNetwork/normalization_multi_input.rst‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
Frame Normalization
2+
===================
3+
4+
This example shows how you can normalize a frame before sending it to another neural network. Many neural network models
5+
require frames with RGB values (pixels) in range between :code:`-0.5` to :code:`0.5`. :ref:`ColorCamera`'s preview outputs
6+
values between :code:`0` and :code:`255`. Simple custom model, created with PyTorch (`link here <https://github.com/luxonis/depthai-experiments/blob/master/gen2-custom-models/generate_model/pytorch_normalize.py>`__, `tutorial here <https://docs.luxonis.com/en/latest/pages/tutorials/creating-custom-nn-models/>`__),
7+
allows users to specify mean and scale factors that will be applied to all frame values (pixels).
8+
9+
.. math::
10+
11+
output = (input - mean) / scale
12+
13+
.. image:: /_static/images/examples/normalize_model.png
14+
15+
On the host, values are converted back to :code:`0`-:code:`255`, so they can be displayed by OpenCV.
16+
17+
.. note::
18+
This is just a demo, for normalization you should use OpenVINO's `model optimizer <https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model_General.html>`__ arguments :code:`--mean_values` and :code:`--scale_values`.
19+
20+
Setup
21+
#####
22+
23+
.. include:: /includes/install_from_pypi.rst
24+
25+
Source code
26+
###########
27+
28+
.. tabs::
29+
30+
.. tab:: Python
31+
32+
Also `available on GitHub <https://github.com/luxonis/depthai-python/blob/main/examples/NeuralNetwork/normalization_multiple_input.py>`__
33+
34+
.. literalinclude:: ../../../../examples/NeuralNetwork/normalization_multi_input.py
35+
:language: python
36+
:linenos:
37+
38+
.. tab:: C++
39+
40+
Also `available on GitHub <https://github.com/luxonis/depthai-core/blob/main/examples/src/normalization_multiple_input.cpp>`__
41+
42+
.. literalinclude:: ../../../../depthai-core/examples/NeuralNetwork/normalization_multi_input.cpp
43+
:language: cpp
44+
:linenos:
45+
46+
.. include:: /includes/footer-short.rst

‎docs/source/tutorials/code_samples.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ Code Samples
1616
../samples/mixed/*
1717
../samples/MobileNet/*
1818
../samples/MonoCamera/*
19+
../samples/NeuralNetwork/*
1920
../samples/ObjectTracker/*
2021
../samples/Script/*
2122
../samples/SpatialDetection/*
@@ -95,6 +96,11 @@ are presented with code.
9596
- :ref:`Mono Camera Control` - Demonstrates how to control the mono camera (crop, exposure, sensitivity) from the host
9697
- :ref:`Mono Full Resolution Saver` - Saves mono (720P) images to the host (:code:`.png`)
9798

99+
.. rubric:: NeuralNetwork
100+
101+
- :ref:`Multi-Input Frame Concat <Multi-Input Frame Concationation>` - Concat mono/rgb streams on the device with a custom model
102+
- :ref:`Frame Normalization` - Normalize the frame on the device with a custom model
103+
98104
.. rubric:: ObjectTracker
99105

100106
- :ref:`Object tracker on video` - Performs object tracking from the video

‎examples/NeuralNetwork/concat_multi_input.py‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
#!/usr/bin/env python3
2+
3+
from pathlib import Path
4+
import sys
5+
import numpy as np
6+
import cv2
7+
import depthai as dai
8+
SHAPE = 300
9+
10+
# Get argument first
11+
nnPath = str((Path(__file__).parent / Path('../models/concat_openvino_2021.4_6shave.blob')).resolve().absolute())
12+
if len(sys.argv) > 1:
13+
nnPath = sys.argv[1]
14+
15+
if not Path(nnPath).exists():
16+
import sys
17+
raise FileNotFoundError(f'Required file/s not found, please run "{sys.executable} install_requirements.py"')
18+
19+
p = dai.Pipeline()
20+
p.setOpenVINOVersion(dai.OpenVINO.VERSION_2021_4)
21+
22+
camRgb = p.createColorCamera()
23+
camRgb.setPreviewSize(SHAPE, SHAPE)
24+
camRgb.setInterleaved(False)
25+
camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
26+
27+
def create_mono(p, socket):
28+
mono = p.create(dai.node.MonoCamera)
29+
mono.setBoardSocket(socket)
30+
mono.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
31+
32+
# ImageManip for cropping (face detection NN requires input image of 300x300) and to change frame type
33+
manip = p.create(dai.node.ImageManip)
34+
manip.initialConfig.setResize(300, 300)
35+
manip.initialConfig.setFrameType(dai.RawImgFrame.Type.BGR888p)
36+
mono.out.link(manip.inputImage)
37+
return manip.out
38+
39+
# NN that detects faces in the image
40+
nn = p.createNeuralNetwork()
41+
nn.setBlobPath(nnPath)
42+
nn.setNumInferenceThreads(2)
43+
44+
camRgb.preview.link(nn.inputs['img2'])
45+
create_mono(p, dai.CameraBoardSocket.LEFT).link(nn.inputs['img1'])
46+
create_mono(p, dai.CameraBoardSocket.RIGHT).link(nn.inputs['img3'])
47+
48+
# Send bouding box from the NN to the host via XLink
49+
nn_xout = p.createXLinkOut()
50+
nn_xout.setStreamName("nn")
51+
nn.out.link(nn_xout.input)
52+
53+
# Pipeline is defined, now we can connect to the device
54+
with dai.Device(p) as device:
55+
qNn = device.getOutputQueue(name="nn", maxSize=4, blocking=False)
56+
shape = (3, SHAPE, SHAPE * 3)
57+
58+
while True:
59+
inNn = np.array(qNn.get().getData())
60+
frame = inNn.view(np.float16).reshape(shape).transpose(1, 2, 0).astype(np.uint8).copy()
61+
62+
cv2.imshow("Concat", frame)
63+
64+
if cv2.waitKey(1) == ord('q'):
65+
break

‎examples/NeuralNetwork/normalization_multi_input.py‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
#!/usr/bin/env python3
2+
3+
from pathlib import Path
4+
import sys
5+
import numpy as np
6+
import cv2
7+
import depthai as dai
8+
SHAPE = 300
9+
10+
# Get argument first
11+
nnPath = str((Path(__file__).parent / Path('../models/normalize_openvino_2021.4_4shave.blob')).resolve().absolute())
12+
if len(sys.argv) > 1:
13+
nnPath = sys.argv[1]
14+
15+
if not Path(nnPath).exists():
16+
import sys
17+
raise FileNotFoundError(f'Required file/s not found, please run "{sys.executable} install_requirements.py"')
18+
19+
p = dai.Pipeline()
20+
p.setOpenVINOVersion(dai.OpenVINO.VERSION_2021_4)
21+
22+
camRgb = p.createColorCamera()
23+
# Model expects values in FP16, as we have compiled it with `-ip FP16`
24+
camRgb.setFp16(True)
25+
camRgb.setInterleaved(False)
26+
camRgb.setPreviewSize(SHAPE, SHAPE)
27+
28+
nn = p.createNeuralNetwork()
29+
nn.setBlobPath(nnPath)
30+
nn.setNumInferenceThreads(2)
31+
32+
script = p.create(dai.node.Script)
33+
script.setScript("""
34+
# Run script only once. We could also send these values from host.
35+
# Model formula:
36+
# output = (input - mean) / scale
37+
38+
# This configuration will subtract all frame values (pixels) by 127.5
39+
# 0.0 .. 255.0 -> -127.5 .. 127.5
40+
data = NNData(2)
41+
data.setLayer("mean", [127.5])
42+
node.io['mean'].send(data)
43+
44+
# This configuration will divide all frame values (pixels) by 255.0
45+
# -127.5 .. 127.5 -> -0.5 .. 0.5
46+
data = NNData(2)
47+
data.setLayer("scale", [255.0])
48+
node.io['scale'].send(data)
49+
""")
50+
51+
# Re-use the initial values for multiplier/addend
52+
script.outputs['mean'].link(nn.inputs['mean'])
53+
nn.inputs['mean'].setWaitForMessage(False)
54+
55+
script.outputs['scale'].link(nn.inputs['scale'])
56+
nn.inputs['scale'].setWaitForMessage(False)
57+
# Always wait for the new frame before starting inference
58+
camRgb.preview.link(nn.inputs['frame'])
59+
60+
# Send normalized frame values to host
61+
nn_xout = p.createXLinkOut()
62+
nn_xout.setStreamName("nn")
63+
nn.out.link(nn_xout.input)
64+
65+
# Pipeline is defined, now we can connect to the device
66+
with dai.Device(p) as device:
67+
qNn = device.getOutputQueue(name="nn", maxSize=4, blocking=False)
68+
shape = (3, SHAPE, SHAPE)
69+
while True:
70+
inNn = np.array(qNn.get().getData())
71+
# Get back the frame. It's currently normalized to -0.5 - 0.5
72+
frame = inNn.view(np.float16).reshape(shape).transpose(1, 2, 0)
73+
# To get original frame back (0-255), we add multiply all frame values (pixels) by 255 and then add 127.5 to them
74+
frame = (frame * 255.0 + 127.5).astype(np.uint8)
75+
# Show the initial frame
76+
cv2.imshow("Original frame", frame)
77+
78+
if cv2.waitKey(1) == ord('q'):
79+
break

‎examples/models/concat-model/model.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Copyright (c) 2021 Intel Corporation
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
description: >-
16+
concat-model
17+
task_type: image_inpainting # Just random task type, so it's supported by model downloader
18+
files:
19+
- name: concat_openvino_2021.4_6shave.blob
20+
size: 1024
21+
sha256: 6ac3023ea8dac9b7501ead0f9b2c2a4495d2791a58b7049de065246455cf87be
22+
source: https://artifacts.luxonis.com/artifactory/luxonis-depthai-data-local/network/concat_openvino_2021.4_6shave.blob
23+
24+
framework: dldt
25+
license: https://raw.githubusercontent.com/openvinotoolkit/open_model_zoo/master/LICENSE

‎examples/models/normalization-model/model.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Copyright (c) 2021 Intel Corporation
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
description: >-
16+
normalization-model
17+
task_type: image_inpainting # Just random task type, so it's supported by model downloader
18+
files:
19+
- name: normalize_openvino_2021.4_4shave.blob
20+
size: 1216
21+
sha256: 9b5dc4b375ed9218c2910284556f8152cbfcc0130e7b7a42d9a9991eae8be23a
22+
source: https://artifacts.luxonis.com/artifactory/luxonis-depthai-data-local/network/normalize_openvino_2021.4_4shave.blob
23+
24+
framework: dldt
25+
license: https://raw.githubusercontent.com/openvinotoolkit/open_model_zoo/master/LICENSE

0 commit comments

Comments
 (0)