Skip to content

Commit 8d6a6d3

Browse files
Erol444Erol444
committed
Added NN examples
1 parent 929a9c1 commit 8d6a6d3

File tree

7 files changed

+285
-0
lines changed

7 files changed

+285
-0
lines changed

‎docs/source/_static/images/examples/normalize_model.png‎

13.6 KB
Loading

‎docs/source/samples/NeuralNetwork/concat_multi_input.rst‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
Multi-Input Frame Concationation
2+
================================
3+
4+
Example concatenates all 3 inputs with a simple custom NN created with PyTorch (`link here <https://github.com/luxonis/depthai-experiments/blob/master/gen2-custom-models/generate_model/pytorch_concat.py>`__, `tutorial here <https://docs.luxonis.com/en/latest/pages/tutorials/creating-custom-nn-models/>`__).
5+
It uses multiple input :ref:`NeuralNetwork` feature and links all 3 camera streams directly to the NN node.
6+
7+
Demo
8+
####
9+
10+
.. image:: https://user-images.githubusercontent.com/18037362/134209980-09c6e2f9-8a26-45d5-a6ad-c31d9e2816e1.png
11+
12+
Setup
13+
#####
14+
15+
.. include:: /includes/install_from_pypi.rst
16+
17+
Source code
18+
###########
19+
20+
.. tabs::
21+
22+
.. tab:: Python
23+
24+
Also `available on GitHub <https://github.com/luxonis/depthai-python/blob/main/examples/NeuralNetwork/concat_multiple_input.py>`__
25+
26+
.. literalinclude:: ../../../../examples/NeuralNetwork/concat_multi_input.py
27+
:language: python
28+
:linenos:
29+
30+
.. tab:: C++
31+
32+
Also `available on GitHub <https://github.com/luxonis/depthai-core/blob/main/examples/src/concat_multiple_input.cpp>`__
33+
34+
.. literalinclude:: ../../../../depthai-core/examples/src/concat_multi_input.cpp
35+
:language: cpp
36+
:linenos:
37+
38+
.. include:: /includes/footer-short.rst

‎docs/source/samples/NeuralNetwork/normalization_multi_input.rst‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
Multi-Input Frame Normalization
2+
===============================
3+
4+
This example shows how you can normalize a frame before sending it to another NN. Many neural network models
5+
require frames with RGB values (pixels) in range between :code:`-0.5` to :code:`0.5`. :ref:`ColorCamera`'s preview outputs
6+
values between :code:`0` and :code:`255`. Simple custom NN, created with PyTorch (`link here <https://github.com/luxonis/depthai-experiments/blob/master/gen2-custom-models/generate_model/pytorch_normalize.py>`__, `tutorial here <https://docs.luxonis.com/en/latest/pages/tutorials/creating-custom-nn-models/>`__),
7+
allows users to specify mean and scale factors that will apply for all frame values (pixels).
8+
9+
.. math::
10+
11+
output = (input - mean) / scale
12+
13+
.. image:: /_static/images/examples/normalize)model.png
14+
15+
On the host, values are converted back to :code:`0`-:code:`255`, so they can be displayed by OpenCV.
16+
17+
.. note::
18+
This is just a demo, for normalization you should look into OpenVINO's `model optimizer <https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model_General.html>`__ arguments :code:`--mean_values` and :code:`--scale_values`.
19+
20+
Setup
21+
#####
22+
23+
.. include:: /includes/install_from_pypi.rst
24+
25+
Source code
26+
###########
27+
28+
.. tabs::
29+
30+
.. tab:: Python
31+
32+
Also `available on GitHub <https://github.com/luxonis/depthai-python/blob/main/examples/NeuralNetwork/normalization_multiple_input.py>`__
33+
34+
.. literalinclude:: ../../../../examples/NeuralNetwork/normalization_multi_input.py
35+
:language: python
36+
:linenos:
37+
38+
.. tab:: C++
39+
40+
Also `available on GitHub <https://github.com/luxonis/depthai-core/blob/main/examples/src/normalization_multiple_input.cpp>`__
41+
42+
.. literalinclude:: ../../../../depthai-core/examples/src/normalization_multi_input.cpp
43+
:language: cpp
44+
:linenos:
45+
46+
.. include:: /includes/footer-short.rst

‎examples/NeuralNetwork/concat_multi_input.py‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
#!/usr/bin/env python3
2+
3+
from pathlib import Path
4+
import sys
5+
import numpy as np
6+
import cv2
7+
import depthai as dai
8+
SHAPE = 300
9+
10+
# Get argument first
11+
nnPath = str((Path(__file__).parent / Path('../models/concat_openvino_2021.4_6shave.blob')).resolve().absolute())
12+
if len(sys.argv) > 1:
13+
nnPath = sys.argv[1]
14+
15+
if not Path(nnPath).exists():
16+
import sys
17+
raise FileNotFoundError(f'Required file/s not found, please run "{sys.executable} install_requirements.py"')
18+
19+
p = dai.Pipeline()
20+
p.setOpenVINOVersion(dai.OpenVINO.VERSION_2021_4)
21+
22+
camRgb = p.createColorCamera()
23+
camRgb.setPreviewSize(SHAPE, SHAPE)
24+
camRgb.setInterleaved(False)
25+
camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
26+
27+
left = p.create(dai.node.MonoCamera)
28+
left.setBoardSocket(dai.CameraBoardSocket.LEFT)
29+
left.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
30+
31+
# ImageManip for cropping (face detection NN requires input image of 300x300) and to change frame type
32+
manipLeft = p.create(dai.node.ImageManip)
33+
manipLeft.initialConfig.setResize(300, 300)
34+
manipLeft.initialConfig.setFrameType(dai.RawImgFrame.Type.BGR888p)
35+
left.out.link(manipLeft.inputImage)
36+
37+
right = p.create(dai.node.MonoCamera)
38+
right.setBoardSocket(dai.CameraBoardSocket.RIGHT)
39+
right.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
40+
41+
# ImageManip for cropping (face detection NN requires input image of 300x300) and to change frame type
42+
manipRight = p.create(dai.node.ImageManip)
43+
manipRight.initialConfig.setResize(300, 300)
44+
manipRight.initialConfig.setFrameType(dai.RawImgFrame.Type.BGR888p)
45+
right.out.link(manipRight.inputImage)
46+
47+
# NN that detects faces in the image
48+
nn = p.createNeuralNetwork()
49+
nn.setBlobPath(nnPath)
50+
nn.setNumInferenceThreads(2)
51+
52+
manipLeft.out.link(nn.inputs['img1'])
53+
camRgb.preview.link(nn.inputs['img2'])
54+
manipRight.out.link(nn.inputs['img3'])
55+
56+
# Send bouding box from the NN to the host via XLink
57+
nn_xout = p.createXLinkOut()
58+
nn_xout.setStreamName("nn")
59+
nn.out.link(nn_xout.input)
60+
61+
# Pipeline is defined, now we can connect to the device
62+
with dai.Device(p) as device:
63+
qNn = device.getOutputQueue(name="nn", maxSize=4, blocking=False)
64+
shape = (3, SHAPE, SHAPE * 3)
65+
66+
while True:
67+
inNn = np.array(qNn.get().getData())
68+
frame = inNn.view(np.float16).reshape(shape).transpose(1, 2, 0).astype(np.uint8).copy()
69+
70+
cv2.imshow("Concat", frame)
71+
72+
if cv2.waitKey(1) == ord('q'):
73+
break

‎examples/NeuralNetwork/normalization_multi_input.py‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
#!/usr/bin/env python3
2+
3+
from pathlib import Path
4+
import sys
5+
import numpy as np
6+
import cv2
7+
import depthai as dai
8+
SHAPE = 300
9+
10+
# Get argument first
11+
nnPath = str((Path(__file__).parent / Path('../models/normalize_openvino_2021.4_4shave.blob')).resolve().absolute())
12+
if len(sys.argv) > 1:
13+
nnPath = sys.argv[1]
14+
15+
if not Path(nnPath).exists():
16+
import sys
17+
raise FileNotFoundError(f'Required file/s not found, please run "{sys.executable} install_requirements.py"')
18+
19+
p = dai.Pipeline()
20+
p.setOpenVINOVersion(dai.OpenVINO.VERSION_2021_4)
21+
22+
camRgb = p.createColorCamera()
23+
# Model expects values in FP16, as we have compiled it with `-ip FP16`
24+
camRgb.setFp16(True)
25+
camRgb.setInterleaved(False)
26+
camRgb.setPreviewSize(SHAPE, SHAPE)
27+
28+
nn = p.createNeuralNetwork()
29+
nn.setBlobPath(nnPath)
30+
nn.setNumInferenceThreads(2)
31+
32+
script = p.create(dai.node.Script)
33+
script.setScript("""
34+
# Run script only once
35+
# Model formula:
36+
# output = (input - mean) / scale
37+
38+
# This configuration will subtract all frame values (pixels) by 127.5
39+
# 0.0 .. 255.0 -> -127.5 .. 127.5
40+
data = NNData(2)
41+
data.setLayer("mean", [127.5])
42+
node.io['mean'].send(data)
43+
44+
# This configuration will divide all frame values (pixels) by 255.0
45+
# -127.5 .. 127.5 -> -0.5 .. 0.5
46+
data = NNData(2)
47+
data.setLayer("scale", [255.0])
48+
node.io['scale'].send(data)
49+
""")
50+
51+
# Re-use the initial values for multiplier/addend
52+
script.outputs['mean'].link(nn.inputs['mean'])
53+
nn.inputs['mean'].setWaitForMessage(False)
54+
55+
script.outputs['scale'].link(nn.inputs['scale'])
56+
nn.inputs['scale'].setWaitForMessage(False)
57+
# Always wait for the new frame before starting inference
58+
camRgb.preview.link(nn.inputs['frame'])
59+
60+
# Send normalized frame values to host
61+
nn_xout = p.createXLinkOut()
62+
nn_xout.setStreamName("nn")
63+
nn.out.link(nn_xout.input)
64+
65+
# Pipeline is defined, now we can connect to the device
66+
with dai.Device(p) as device:
67+
qNn = device.getOutputQueue(name="nn", maxSize=4, blocking=False)
68+
shape = (3, SHAPE, SHAPE)
69+
while True:
70+
inNn = np.array(qNn.get().getData())
71+
# Get back the frame. It's currently normalized to -1.0 - 1.0
72+
frame = inNn.view(np.float16).reshape(shape).transpose(1, 2, 0)
73+
# To get original frame back (0-255), we add multiply all frame values (pixels) by 255 and then add 127.5 to them.
74+
frame = (frame * 255.0 + 127.5).astype(np.uint8)
75+
cv2.imshow("Original frame", frame)
76+
77+
if cv2.waitKey(1) == ord('q'):
78+
break

‎examples/models/concat-model/model.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Copyright (c) 2021 Intel Corporation
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
description: >-
16+
concat-model
17+
task_type: image_inpainting # Just random task type, so it's supported by model downloader
18+
files:
19+
- name: concat_openvino_2021.4_6shave.blob
20+
size: 1024
21+
sha256: 6ac3023ea8dac9b7501ead0f9b2c2a4495d2791a58b7049de065246455cf87be
22+
source: https://artifacts.luxonis.com/artifactory/luxonis-depthai-data-local/network/concat_openvino_2021.4_6shave.blob
23+
24+
framework: dldt
25+
license: https://raw.githubusercontent.com/openvinotoolkit/open_model_zoo/master/LICENSE

‎examples/models/normalization-model/model.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Copyright (c) 2021 Intel Corporation
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
description: >-
16+
normalization-model
17+
task_type: image_inpainting # Just random task type, so it's supported by model downloader
18+
files:
19+
- name: normalize_openvino_2021.4_4shave.blob
20+
size: 1216
21+
sha256: 9b5dc4b375ed9218c2910284556f8152cbfcc0130e7b7a42d9a9991eae8be23a
22+
source: https://artifacts.luxonis.com/artifactory/luxonis-depthai-data-local/network/normalize_openvino_2021.4_4shave.blob
23+
24+
framework: dldt
25+
license: https://raw.githubusercontent.com/openvinotoolkit/open_model_zoo/master/LICENSE

0 commit comments

Comments
 (0)