Getting Started With Oak-D-Lite on Create3

[slowrunner]

Wow, oh WOW! The Oak-D-Lite is an impressive Computer Vision "Swiss Army Knife". Everything can be (and must be) configured, so every person's use of the device is likely to be different.

My goals for the Oak-D-Lite on my Create3 "WaLI" robot are:

• Publish ROS topic equivalent of "Detected Object at ROS coordinates X,Y,Z"
• Capture still frame (Full Color, Max Resolution, Full Sensor Aspect Ratio 16:9)
• Capture still frame (300x300) for input to eventual off-board transfer learning
• Publish ROS topics needed for input to RTABmap:
  • for RTABmap mapping
  • for RTABmap localization
  • for RTAbmap to generate dynamic cost map (for Nav2 navigation)

My path to this is:

• Check that Oak-D-Lite works with the python depthai example: spatial tiny yolo v4 running native on the Raspberry Pi5
• Check that Oak-D-Lite works from inside my ROS2 Humble Docker container
• Install Luxonis Docker Depthai-ROS driver, configure for Oak-D-Lite and object detection, configure rviz2 display depthai topics

I have the first two steps down and have started on the ROS-Driver step.

Results from initial tests:

**Oak-D-Lite POWER CONSUMPTON With Console Only Spatial Tiny Yolo**
(Wali is a set of ROS 2 Humble robot nodes over Ubuntu 22.04 in a Docker container)

- Base Pi5 running Wali 3.3W (Oak-D-Lite unplugged)
- Pi5 running Wali plus Oak-D-Lite first plugged in: 3.95W (0.65W for Oak-D-Lite powered, non-running state)
- Pi5 running Wali plus console spatial tiny yolo.py 28fps: 8.05W (4.75W for Oak-D-Lite processing)
- Pi5 running Wali after console spatial tiny yolo.py shutdown: 4.18W (0.9W for Oak-D-Lite powered, non-running)

(vdepthai)pi@WaLiPi5:~/wali_pi5/systests/Oak-D-Lite/vdepthai/spatial_tiny_yolo $./console_spatial_tiny_yolo.py 

NN fps: 28.3     
person     conf:2 %  at X:17     Y:27     Z:628   mm 

NN fps: 28.3     
person     conf:7 %  at X:11     Y:35     Z:628   mm  

NN fps: 28.3   
person     conf:8 %  at X:11     Y:68     Z:628   mm
tie        conf:3 %  at X:23     Y:50     Z:628   mm

Holding a framed picture of me in a tuxedo (bow tie) in the FOV of Oak-D-Lite.

[slowrunner]

WOW, WOW, WOW - It worked!

I am stunned - I managed:

• to get the ROS 2 Humble docker depthai-ros container running, and networked with my bot and my desktop visualization rviz2

• to launch the "mobilenet publisher" example (successfully)

• to visualize the /color/image topic in rviz2 - WOW, WOW, WOW that is exciting
  Confirmed image of bicycle or person are centered in detection field of view

• to echo the /color/mobilenet_detections topic - Another wow!

results:  
  - hypothesis:
      class_id: '15'
      score: 0.849609375

results:
  - hypothesis:
      class_id: '2'
      score: 0.9404296875

BUT - where is the class id list? (Docs do not give a clue… I FOUND IT…)

        /depthai/resources/nn/mobilenet-ssd/mobilenet-ssd.json"labels":

        [            
"background",             
"aeroplane",            
"bicycle",            
"bird",            
"boat",            
"bottle",            
"bus",            
"car",            
"cat",           
 "chair",            
"cow",           
 "diningtable",            
"dog",            
"horse",            
"motorbike",            
"person",            
"pottedplant",            
"sheep",           
 "sofa",            
"train",            
"tvmonitor"        ]

So 2 is the bicycle I put in front of the camera,
and 15 is the person I put in front of the camera.

And by adding ros-humble-depthai-ros to my docker file, voila I can launch the Oak-D-Light ros node!

Now to write my first Create3 behavior node using the Oak-D-Light running mobilenet-ssd neural network object recognition (I'm not bringing any cows into the house to test it though).