Merge pull request #16 from NVIDIA-ISAAC-ROS/release-dp3

Isaac ROS 0.30.0 (DP3)

Author: jaiveersinghNV

.gitattributes

@@ -1,3 +1,6 @@
+# Ignore Python files in linguist
+*.py linguist-detectable=false
+
 # Images
 *.gif filter=lfs diff=lfs merge=lfs -text
 *.jpg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -4,19 +4,29 @@
 
 ## Overview
 
-This repository provides NVIDIA GPU-accelerated packages for 3D object pose estimation. Using a deep learned pose estimation model and a monocular camera, the `isaac_ros_dope` and `isaac_ros_centerpose` package can estimate the 6DOF pose of a target object.
+Isaac ROS Pose Estimation contains ROS 2 packages to predict the pose of an object. `isaac_ros_dope` provides a pose estimation method using 3D bounding cuboid dimensions of a known object in an input image. `isaac_ros_centerpose` provides a pose estimation method using 3D bounding cuboid dimensions of unknown object instances in a known category of objects from an input image. `isaac_ros_dope` and `isaac_ros_centerpose` use GPU acceleration for DNN inference to estimate the pose of an object. The output prediction can be used by perception functions when fusing with a corresponding depth to provide the 3D pose of an object and distance for navigation or manipulation.
+
+<div align="center"><img src="resources/isaac_ros_pose_estimation_nodegraph.png" width="500px"/></div>
+
+`isaac_ros_dope` is used in a graph of nodes to estimate the pose of a known object with 3D bounding cuboid dimensions. To produce the estimate, a [DOPE](https://github.com/NVlabs/Deep_Object_Pose) (Deep Object Pose Estimation) pre-trained model is required. Input images may need to be cropped and resized to maintain the aspect ratio and match the input resolution of DOPE. After DOPE has produced an estimate, the DNN decoder will use the specified object type to transform using belief maps to output object poses.
+
+NVLabs has provided a DOPE pre-trained model using the [HOPE](https://github.com/swtyree/hope-dataset) dataset. HOPE stands for household objects for pose estimation and is a research-oriented dataset using toy grocery objects and 3D textured meshes of the objects for training on synthetic data. To use DOPE for other objects that are relevant to your application, it needs to be trained with another dataset targeting these objects. For example, DOPE has been trained to detect dollies for use with a mobile robot that navigates under, lifts, and moves that type of dolly.
+
+`isaac_ros_centerpose` has similarities to `isaac_ros_dope` in that both estimate an object pose; however, `isaac_ros_centerpose` provides additional functionality. The [CenterPose](https://github.com/NVlabs/CenterPose) DNN performs object detection on the image, generates 2D keypoints for the object, estimates the 6-DoF pose, and regresses relative 3D bounding cuboid dimensions. This is performed on a known object class without knowing the instance--for example, detecting a chair without having trained on images of all chairs. NVLabs has provided pre-trained models for the CenterPose model; however, as with the DOPE model, it needs to be trained with another dataset targeting objects that are specific to your application.  
+
+Pose estimation is a compute-intensive task and not performed at the frame rate of an input camera. To make efficient use of resources, object pose is estimated for a single frame and used as an input to navigation. Additional object pose estimates are computed to further refine navigation in progress at a lower frequency than the input rate of a typical camera.
 
 Packages in this repository rely on accelerated DNN model inference using [Triton](https://github.com/triton-inference-server/server) or [TensorRT](https://developer.nvidia.com/tensorrt) from [Isaac ROS DNN Inference](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_inference).
 
 ## Performance
 
-The following are the benchmark performance results of the prepared pipelines in this package, by supported platform:
+The following table summarizes the per-platform performance statistics of sample graphs that use this package, with links included to the full benchmark output. These benchmark configurations are taken from the [Isaac ROS Benchmark](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark#list-of-isaac-ros-benchmarks) collection, based on the [`ros2_benchmark`](https://github.com/NVIDIA-ISAAC-ROS/ros2_benchmark) framework.
 
-| Pipeline     | AGX Orin         | Orin Nano | x86_64 w/ RTX3060  |
-| ------------ | ---------------- | --------- | ------------------ |
-| `DOPE` (VGA) | 40 fps <br> 40ms | N/A       | 84 fps <br> 15.4ms |
+| Sample Graph                                                                                                                                         | Input Size | AGX Orin                                                                                                                                        | Orin NX                                                                                                                                        | Orin Nano 8GB                                                                                                                                        | x86_64 w/ RTX 3060 Ti                                                                                                                                    |
+| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [DOPE Pose Estimation Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/scripts//isaac_ros_dope_graph.py)             | VGA        | [40.4 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_dope_graph-agx_orin.json)<br>29 ms       | [16.7 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_dope_graph-orin_nx.json)<br>120 ms      | --                                                                                                                                                   | [82.8 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_dope_graph-x86_64_rtx_3060Ti.json)<br>14 ms       |
+| [Centerpose Pose Estimation Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/scripts//isaac_ros_centerpose_graph.py) | VGA        | [50.2 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_centerpose_graph-agx_orin.json)<br>38 ms | [20.2 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_centerpose_graph-orin_nx.json)<br>67 ms | [15.2 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_centerpose_graph-orin_nano_8gb.json)<br>85 ms | [13.5 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_centerpose_graph-x86_64_rtx_3060Ti.json)<br>42 ms |
 
-These data have been collected per the methodology described [here](https://github.com/NVIDIA-ISAAC-ROS/.github/blob/main/profile/performance-summary.md#methodology).
 
 ## Table of Contents
 
@@ -53,24 +63,24 @@ These data have been collected per the methodology described [here](https://gith
 
 ## Latest Update
 
-Update 2022-10-19: Updated OSS licensing
+Update 2023-04-05: Source available GXF extensions
 
 ## Supported Platforms
 
-This package is designed and tested to be compatible with ROS2 Humble running on [Jetson](https://developer.nvidia.com/embedded-computing) or an x86_64 system with an NVIDIA GPU.
+This package is designed and tested to be compatible with ROS 2 Humble running on [Jetson](https://developer.nvidia.com/embedded-computing) or an x86_64 system with an NVIDIA GPU.
 
-> **Note**: Versions of ROS2 earlier than Humble are **not** supported. This package depends on specific ROS2 implementation features that were only introduced beginning with the Humble release.
+> **Note**: Versions of ROS 2 earlier than Humble are **not** supported. This package depends on specific ROS 2 implementation features that were only introduced beginning with the Humble release.
 
-| Platform | Hardware                                                                                                                                                                                                 | Software                                                                                                             | Notes                                                                                                                                                                                   |
-| -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Jetson   | [Jetson Orin](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/) <br> [Jetson Xavier](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-agx-xavier/) | [JetPack 5.0.2](https://developer.nvidia.com/embedded/jetpack)                                                       | For best performance, ensure that [power settings](https://docs.nvidia.com/jetson/archives/r34.1/DeveloperGuide/text/SD/PlatformPowerAndPerformance.html) are configured appropriately. |
-| x86_64   | NVIDIA GPU                                                                                                                                                                                               | [Ubuntu 20.04+](https://releases.ubuntu.com/20.04/) <br> [CUDA 11.6.1+](https://developer.nvidia.com/cuda-downloads) |
+| Platform | Hardware                                                                                                                                                                                                 | Software                                                                                                           | Notes                                                                                                                                                                                   |
+| -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Jetson   | [Jetson Orin](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/) <br> [Jetson Xavier](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-agx-xavier/) | [JetPack 5.1.1](https://developer.nvidia.com/embedded/jetpack)                                                     | For best performance, ensure that [power settings](https://docs.nvidia.com/jetson/archives/r34.1/DeveloperGuide/text/SD/PlatformPowerAndPerformance.html) are configured appropriately. |
+| x86_64   | NVIDIA GPU                                                                                                                                                                                               | [Ubuntu 20.04+](https://releases.ubuntu.com/20.04/) <br> [CUDA 11.8+](https://developer.nvidia.com/cuda-downloads) |
 
 ### Docker
 
 To simplify development, we strongly recommend leveraging the Isaac ROS Dev Docker images by following [these steps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common/blob/main/docs/dev-env-setup.md). This will streamline your development environment setup with the correct versions of dependencies on both Jetson and x86_64 platforms.
 
-> **Note:** All Isaac ROS Quickstarts, tutorials, and examples have been designed with the Isaac ROS Docker images as a prerequisite.
+> **Note**: All Isaac ROS Quickstarts, tutorials, and examples have been designed with the Isaac ROS Docker images as a prerequisite.
 
 ## Quickstart
 
@@ -99,6 +109,10 @@ To simplify development, we strongly recommend leveraging the Isaac ROS Dev Dock
     git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_inference
     ```
 
+    ```bash
+    git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_pipeline
+    ```
+
 3. Pull down a ROS Bag of sample data:
 
     ```bash
@@ -222,16 +236,17 @@ To simplify development, we strongly recommend leveraging the Isaac ROS Dev Dock
 
     <div align="center"><img src="resources/dope_rviz2.png" width="600px"/></div>
 
-    > **Note:** For best results, crop or resize input images to the same dimensions your DNN model is expecting.
+    > **Note**: For best results, crop or resize input images to the same dimensions your DNN model is expecting.
 
 ## Next Steps
 
 ### Try More Examples
 
 To continue your exploration, check out the following suggested examples:
 
-- Using `DOPE` with `Triton` can be found [here](docs/dope-triton.md)
-- Using `Centerpose` with `Triton` can be found [here](docs/centerpose.md)
+- [`DOPE` with `Triton`](docs/dope-triton.md)
+- [`Centerpose` with `Triton`](docs/centerpose.md)
+- [`DOPE` with non-standard input image sizes](docs/dope-custom-size.md)
 
 ### Use Different Models
 
@@ -356,6 +371,7 @@ For solutions to problems with using DNN models, please check [here](https://git
 
 | Date       | Changes                                                                                                  |
 | ---------- | -------------------------------------------------------------------------------------------------------- |
+| 2023-04-05 | Source available GXF extensions                                                                          |
 | 2022-06-30 | Update to use NITROS for improved performance and to be compatible with JetPack 5.0.2                    |
 | 2022-06-30 | Refactored README, updated launch file & added `nvidia` namespace, dropped Jetson support for CenterPose |
 | 2021-10-20 | Initial update                                                                                           |

docs/centerpose.md

@@ -4,7 +4,7 @@
 
 ## Overview
 
-This tutorial walks you through a pipeline to estimate the 6DOF pose of a target object using [CenterPose](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation) with Triton. It uses input monocular images from a rosbag.
+This tutorial walks you through a graph to estimate the 6DOF pose of a target object using [CenterPose](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation) with Triton. It uses input monocular images from a rosbag.
 > **Warning**: These steps will only work on `x86_64` and **NOT** on `Jetson`.
 
 ## Tutorial Walkthrough

docs/dope-custom-size.md

@@ -0,0 +1,26 @@
+# Using DOPE with a non-standard input image size
+
+## Overview
+
+The DOPE network architecture, as outlined in the [original paper](https://arxiv.org/abs/1809.10790), can receive input images of arbitrary size and subsequently produce output belief maps of the corresponding dimensions.
+
+However, the ONNX format used to run this network on Triton or TensorRT is not as flexible, and an ONNX-exported model **does NOT** support arbitrary image sizes at inference time. Instead, the desired input image dimensions must be explicitly specified when preparing the ONNX file using the `dope_converter.py` script, as referenced in the [quickstart](../README.md#quickstart).
+
+## Tutorial Walkthrough
+
+1. Follow steps 1-6 of the main DOPE [quickstart](../README.md#quickstart).
+
+2. At step 7, run the `dope_converter.py` script with the two additional arguments `row` and `col` specifying the desired input image size:
+
+    ```bash
+    python3 /workspaces/isaac_ros-dev/src/isaac_ros_pose_estimation/isaac_ros_dope/scripts/dope_converter.py --format onnx --input /tmp/models/Ketchup.pth --row 1080 --col 1920
+    ```
+
+3. Proceed through steps 8-9.
+4. At step 10, launch the ROS 2 launchfile with two additional arguments `network_image_height` and `network_image_width` specifying the desired input image size:
+
+    ```bash
+    ros2 launch isaac_ros_dope isaac_ros_dope_tensor_rt.launch.py model_file_path:=/tmp/models/Ketchup.onnx engine_file_path:=/tmp/models/Ketchup.plan network_image_height:=1080 network_image_width:=1920
+    ```
+
+5. Continue with the rest of the quickstart. You should now be able to detect poses in images of your desired size.

docs/dope-triton.md

@@ -4,7 +4,7 @@
 
 ## Overview
 
-This tutorial walks you through a pipeline to estimate the 6DOF pose of a target object using [DOPE](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation) using different backends. It uses input monocular images from a rosbag. The different backends show are:
+This tutorial walks you through a graph to estimate the 6DOF pose of a target object using [DOPE](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation) using different backends. It uses input monocular images from a rosbag. The different backends show are:
 
 1. PyTorch and ONNX
 2. TensorRT Plan files with Triton
@@ -70,8 +70,8 @@ This tutorial walks you through a pipeline to estimate the 6DOF pose of a target
 
     The `<insert-platform>` part should be replaced with `onnxruntime_onnx` for `.onnx` files, `tensorrt_plan` for `.plan` files and `pytorch_libtorch` for `.pt` files.
 
-    > **Note**: The DOPE decoder currently works with the output of a DOPE network that has a fixed input size of 640 x 480, which are the default dimensions set in the script. In order to use input images of other sizes, make sure to crop or resize using ROS2 nodes from [Isaac ROS Image Pipeline](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_pipeline) or similar packages.
-
+    > **Note**: The DOPE decoder currently works with the output of a DOPE network that has a fixed input size of 640 x 480, which are the default dimensions set in the script. In order to use input images of other sizes, make sure to crop or resize using ROS 2 nodes from [Isaac ROS Image Pipeline](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_pipeline) or similar packages.
+<!-- Split blockquote -->
     > **Note**: The model name must be `model.onnx`.
 
 5. Rebuild and source `isaac_ros_dope`:
@@ -125,4 +125,4 @@ This tutorial walks you through a pipeline to estimate the 6DOF pose of a target
 
     Then click on the `Add` button, select `By topic` and choose `PoseArray` under `/poses`. Finally, change the display to show an axes by updating `Shape` to be `Axes`, as shown in the screenshot at the top of this page. Make sure to update the `Fixed Frame` to `camera`.
 
-    > **Note:** For best results, crop/resize input images to the same dimensions your DNN model is expecting.
+    > **Note**: For best results, crop/resize input images to the same dimensions your DNN model is expecting.