Merge pull request #24 from NVIDIA-ISAAC-ROS/release-3.0

Isaac ROS 3.0.1

Author: jaiveersinghNV

README.md

@@ -15,24 +15,40 @@ Learn how to use this package by watching our on-demand webinar:
 
 ## Overview
 
-[Isaac ROS DNN Stereo Depth](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_stereo_depth) provides a GPU-accelerated package for DNN-based
-stereo disparity. Stereo disparity is calculated from a
-time-synchronized image pair sourced from a stereo camera and is used to
-produce a depth image or a point cloud for a scene. The `isaac_ros_ess`
-package uses the [ESS DNN
-model](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/models/dnn_stereo_disparity)
-to perform stereo depth estimation via continuous disparity prediction.
-Given a pair of stereo input images, the package generates a disparity
-map of the left input image.
+The vision depth perception problem is generally useful in many fields of robotics such as estimating
+the pose of a robotic arm in an object manipulation task, estimating distance of static or moving targets
+in autonomous robot navigation, tracking targets in delivery robots and so on.
+[Isaac ROS DNN Stereo Depth](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_stereo_depth) is targeted at two Isaac applications,
+Isaac Manipulator and Isaac Perceptor. In Isaac Manipulator application, ESS is deployed in
+[Isaac ROS cuMotion](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_cumotion/index.html)
+package as a plug-in node to provide depth perception maps for robot arm motion planning and control.
+In this scenario, multi-camera stereo streams of industrial robot arms on a table task are passed to ESS to
+obtain corresponding depth streams. The depth streams are used to segment the relative distance of robot arms from
+corresponding objects on the table; thus providing signals for collision avoidance and fine-grain control.
+Similarly, the Isaac Perceptor application uses several Isaac ROS packages, namely,
+[Isaac ROS Nova](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_nova/index.html),
+[Isaac ROS Visual Slam](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_visual_slam/index.html),
+[Isaac ROS Stereo Depth (ESS)](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_dnn_stereo_depth/index.html),
+[Isaac ROS Nvblox](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_nvblox/index.html)
+and [Isaac ROS Image Pipeline](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_pipeline/index.html).
+
+ESS is deployed in Isaac Perceptor to enable Nvblox to create
+3D voxelized images of the robot surroundings. Specifically, the Nova developer suite provides 3x stereo-camera
+streams to Isaac Perceptor. Each stream corresponds to the front, left, and right cameras.
+In both Isaac Manipulator and Isaac Perceptor, a camera-specific image processing pipeline consisting of
+GPU-accelerated operations, provides rectification and undistortion of the input stereo images. All stereo stream image
+pair are time synchronized before before passing them to ESS. ESS node outputs corresponding depth maps for all three
+preprocessed image streams and combines the depth images with motion signals provided by cuVSLAM module.
+The combined depth and motion integrated signals are fed to Nvblox module to produce a dense 3D volumetric scene
+reconstruction of the surrounding scene.
 
 <div align="center"><a class="reference internal image-reference" href="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_dnn_stereo_depth/isaac_ros_ess_nodegraph.png/"><img alt="image" src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_dnn_stereo_depth/isaac_ros_ess_nodegraph.png/" width="800px"/></a></div>
 
-ESS is used in a graph of nodes to provide a disparity prediction from an input left and right stereo image pair.
-Images to ESS need to be rectified and resized to the appropriate input resolution.
-The aspect ratio of the image is recommended to be maintained, so the image may need to be cropped and resized to maintain the input aspect ratio.
-The graph for DNN encode, DNN inference, and DNN decode is included in the ESS node.
-Inference is performed using TensorRT, as the ESS DNN model is designed with optimizations supported by TensorRT.
-ESS node is agnostic to the model dimension and disparity output has the same dimension as the ESS model.
+Above, ESS node is used in a graph of nodes to provide a disparity prediction from an input left and right stereo image pair.
+The rectify and resize nodes pre-process the left and right frames to the appropriate resolution.
+The aspect ratio of the image is recommended to be maintained to avoid degrading the depth output quality.
+The graph for DNN encode, DNN inference, and DNN decode is included in the ESS node. Inference is performed using
+TensorRT, as the ESS DNN model is designed with optimizations supported by TensorRT.
 
 ## Isaac ROS NITROS Acceleration
 

gxf_isaac_ess/package.xml

@@ -21,7 +21,7 @@ SPDX-License-Identifier: Apache-2.0
 <?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
 <package format="3">
   <name>gxf_isaac_ess</name>
-  <version>3.0.0</version>
+  <version>3.0.1</version>
   <description>Disparity GXF extension.</description>
 
   <maintainer email="[email protected]">Isaac ROS Maintainers</maintainer>

gxf_isaac_video_buffer_utils/package.xml

@@ -21,7 +21,7 @@ SPDX-License-Identifier: Apache-2.0
 <?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
 <package format="3">
   <name>gxf_isaac_video_buffer_utils</name>
-  <version>3.0.0</version>
+  <version>3.0.1</version>
   <description>GXF extension containing miscellaneous utility components for video buffer.</description>
 
   <maintainer email="[email protected]">Isaac ROS Maintainers</maintainer>

isaac_ros_ess/config/isaac_ros_ess_isaac_sim.rviz

@@ -7,7 +7,7 @@ Panels:
         - /Global Options1
         - /Status1
       Splitter Ratio: 0.5
-    Tree Height: 335
+    Tree Height: 352
   - Class: rviz_common/Selection
     Name: Selection
   - Class: rviz_common/Tool Properties
@@ -93,7 +93,7 @@ Visualization Manager:
         Durability Policy: Volatile
         History Policy: Keep Last
         Reliability Policy: Reliable
-        Value: /front_stereo_camera/left_rgb/image_resize
+        Value: /front_stereo_camera/left/image_resize
       Value: true
     - Class: rviz_default_plugins/Image
       Enabled: true
@@ -107,7 +107,7 @@ Visualization Manager:
         Durability Policy: Volatile
         History Policy: Keep Last
         Reliability Policy: Reliable
-        Value: /front_stereo_camera/left_rgb/image_resize
+        Value: /front_stereo_camera/right/image_resize
       Value: true
   Enabled: true
   Global Options:
@@ -178,12 +178,12 @@ Visualization Manager:
 Window Geometry:
   Displays:
     collapsed: false
-  Height: 1066
+  Height: 1099
   Hide Left Dock: false
   Hide Right Dock: false
   Left Image:
     collapsed: false
-  QMainWindow State: 000000ff00000000fd0000000400000000000001b300000390fc020000000afb0000001200530065006c0065006300740069006f006e00000001e10000009b0000005c00fffffffb0000001e0054006f006f006c002000500072006f007000650072007400690065007302000001ed000001df00000185000000a3fb000000120056006900650077007300200054006f006f02000001df000002110000018500000122fb000000200054006f006f006c002000500072006f0070006500720074006900650073003203000002880000011d000002210000017afb000000100044006900730070006c006100790073010000003b000001d8000000c700fffffffb0000002000730065006c0065006300740069006f006e00200062007500660066006500720200000138000000aa0000023a00000294fb00000014005700690064006500530074006500720065006f02000000e6000000d2000003ee0000030bfb0000000c004b0069006e0065006300740200000186000001060000030c00000261fb00000014004c00650066007400200049006d0061006700650100000219000000d90000002800fffffffb000000160052006900670068007400200049006d00610067006501000002f8000000d30000002800ffffff000000010000010000000390fc0200000003fb0000001e0054006f006f006c002000500072006f00700065007200740069006500730100000041000000780000000000000000fb0000000a00560069006500770073010000003b00000390000000a000fffffffb0000001200530065006c0065006300740069006f006e010000025a000000b200000000000000000000000200000490000000a9fc0100000001fb0000000a00560069006500770073030000004e00000080000002e10000019700000003000007800000003efc0100000002fb0000000800540069006d00650100000000000007800000024400fffffffb0000000800540069006d00650100000000000004500000000000000000000004c10000039000000004000000040000000800000008fc0000000100000002000000010000000a0054006f006f006c00730100000000ffffffff0000000000000000
+  QMainWindow State: 000000ff00000000fd0000000400000000000001b3000003b1fc020000000afb0000001200530065006c0065006300740069006f006e00000001e10000009b0000005c00fffffffb0000001e0054006f006f006c002000500072006f007000650072007400690065007302000001ed000001df00000185000000a3fb000000120056006900650077007300200054006f006f02000001df000002110000018500000122fb000000200054006f006f006c002000500072006f0070006500720074006900650073003203000002880000011d000002210000017afb000000100044006900730070006c006100790073010000003b000001e9000000c700fffffffb0000002000730065006c0065006300740069006f006e00200062007500660066006500720200000138000000aa0000023a00000294fb00000014005700690064006500530074006500720065006f02000000e6000000d2000003ee0000030bfb0000000c004b0069006e0065006300740200000186000001060000030c00000261fb00000014004c00650066007400200049006d006100670065010000022a000000e10000002800fffffffb000000160052006900670068007400200049006d0061006700650100000311000000db0000002800ffffff0000000100000100000003b1fc0200000003fb0000001e0054006f006f006c002000500072006f00700065007200740069006500730100000041000000780000000000000000fb0000000a00560069006500770073010000003b000003b1000000a000fffffffb0000001200530065006c0065006300740069006f006e010000025a000000b200000000000000000000000200000490000000a9fc0100000001fb0000000a00560069006500770073030000004e00000080000002e10000019700000003000007800000003efc0100000002fb0000000800540069006d00650100000000000007800000025300fffffffb0000000800540069006d00650100000000000004500000000000000000000004c1000003b100000004000000040000000800000008fc0000000100000002000000010000000a0054006f006f006c00730100000000ffffffff0000000000000000
   Right Image:
     collapsed: false
   Selection:
@@ -196,4 +196,4 @@ Window Geometry:
     collapsed: false
   Width: 1920
   X: 0
-  Y: 267
+  Y: 240

isaac_ros_ess/launch/isaac_ros_ess_isaac_sim.launch.py

@@ -50,10 +50,10 @@ def generate_launch_description():
                 'encoding_desired': 'rgb8',
         }],
         remappings=[
-            ('camera_info', 'front_stereo_camera/right_rgb/camerainfo'),
-            ('image', 'front_stereo_camera/right_rgb/image_raw'),
-            ('resize/camera_info', 'front_stereo_camera/right_rgb/camerainfo_resize'),
-            ('resize/image', 'front_stereo_camera/right_rgb/image_resize')]
+            ('camera_info', 'front_stereo_camera/right/camera_info'),
+            ('image', 'front_stereo_camera/right/image_rect_color'),
+            ('resize/camera_info', 'front_stereo_camera/right/camera_info_resize'),
+            ('resize/image', 'front_stereo_camera/right/image_resize')]
     )
 
     image_resize_node_left = ComposableNode(
@@ -66,10 +66,10 @@ def generate_launch_description():
                 'encoding_desired': 'rgb8',
         }],
         remappings=[
-            ('camera_info', 'front_stereo_camera/left_rgb/camerainfo'),
-            ('image', 'front_stereo_camera/left_rgb/image_raw'),
-            ('resize/camera_info', 'front_stereo_camera/left_rgb/camerainfo_resize'),
-            ('resize/image', 'front_stereo_camera/left_rgb/image_resize')]
+            ('camera_info', 'front_stereo_camera/left/camera_info'),
+            ('image', 'front_stereo_camera/left/image_rect_color'),
+            ('resize/camera_info', 'front_stereo_camera/left/camera_info_resize'),
+            ('resize/image', 'front_stereo_camera/left/image_resize')]
     )
 
     disparity_node = ComposableNode(
@@ -80,11 +80,11 @@ def generate_launch_description():
                      'threshold': threshold,
                      'input_layer_width': 960,
                      'input_layer_height': 576}],
-        remappings=[('left/image_rect', 'front_stereo_camera/left_rgb/image_resize'),
+        remappings=[('left/image_rect', 'front_stereo_camera/left/image_resize'),
                     ('left/camera_info',
-                     'front_stereo_camera/left_rgb/camerainfo_resize'),
-                    ('right/image_rect', 'front_stereo_camera/right_rgb/image_resize'),
-                    ('right/camera_info', 'front_stereo_camera/right_rgb/camerainfo_resize')])
+                     'front_stereo_camera/left/camera_info_resize'),
+                    ('right/image_rect', 'front_stereo_camera/right/image_resize'),
+                    ('right/camera_info', 'front_stereo_camera/right/camera_info_resize')])
 
     pointcloud_node = ComposableNode(
         package='isaac_ros_stereo_image_proc',
@@ -93,10 +93,10 @@ def generate_launch_description():
             'use_color': True,
             'unit_scaling': 1.0
         }],
-        remappings=[('left/image_rect_color', 'front_stereo_camera/left_rgb/image_resize'),
+        remappings=[('left/image_rect_color', 'front_stereo_camera/left/image_resize'),
                     ('left/camera_info',
-                     'front_stereo_camera/left_rgb/camerainfo_resize'),
-                    ('right/camera_info', 'front_stereo_camera/right_rgb/camerainfo_resize')])
+                     'front_stereo_camera/left/camera_info_resize'),
+                    ('right/camera_info', 'front_stereo_camera/right/camera_info_resize')])
 
     container = ComposableNodeContainer(
         name='disparity_container',

isaac_ros_ess/package.xml

@@ -21,7 +21,7 @@ SPDX-License-Identifier: Apache-2.0
 <?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
 <package format="3">
   <name>isaac_ros_ess</name>
-  <version>3.0.0</version>
+  <version>3.0.1</version>
   <description>DNN Stereo Disparity Network for Isaac ROS</description>
 
   <maintainer email="[email protected]">Isaac ROS Maintainers</maintainer>
@@ -42,7 +42,7 @@ SPDX-License-Identifier: Apache-2.0
   <depend>gxf_isaac_messages</depend>
   <depend>gxf_isaac_messages_throttler</depend>
   <depend>isaac_ros_ess_models_install</depend>
-  
+
   <build_depend>isaac_ros_common</build_depend>
   <build_depend>gxf_isaac_gems</build_depend>
 

isaac_ros_ess_models_install/package.xml

@@ -21,7 +21,7 @@ SPDX-License-Identifier: Apache-2.0
 <?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
 <package format="3">
   <name>isaac_ros_ess_models_install</name>
-  <version>3.0.0</version>
+  <version>3.0.1</version>
   <description>Scripts for installing ESS models</description>
 
   <maintainer email="[email protected]">Isaac ROS Maintainers</maintainer>