Skip to content

Commit 1b0b17b

Browse files
jaiveersinghNVjaiveersinghNV
committed
Isaac ROS 0.30.0 (DP3)
1 parent 6ab7e5d commit 1b0b17b

File tree

70 files changed

+5517
-194
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

70 files changed

+5517
-194
lines changed

‎.gitattributes‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
# Ignore Python files in linguist
2+
*.py linguist-detectable=false
3+
14
# Images
25
*.gif filter=lfs diff=lfs merge=lfs -text
36
*.jpg filter=lfs diff=lfs merge=lfs -text
@@ -16,5 +19,10 @@
1619
*.so filter=lfs diff=lfs merge=lfs -text
1720
*.so.* filter=lfs diff=lfs merge=lfs -text
1821

22+
# ROS Bags
23+
**/resources/**/*.zstd filter=lfs diff=lfs merge=lfs -text
24+
**/resources/**/*.db3 filter=lfs diff=lfs merge=lfs -text
25+
**/resources/**/*.yaml filter=lfs diff=lfs merge=lfs -text
26+
1927
# Model files
20-
*.onnx filter=lfs diff=lfs merge=lfs -text
28+
*.onnx filter=lfs diff=lfs merge=lfs -text

‎README.md‎

Lines changed: 45 additions & 27 deletions
Large diffs are not rendered by default.

‎docs/tensorrt-and-triton-info.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,13 +28,13 @@ Users can either prepare a custom model or download pre-trained models from NGC
2828

2929
In order to be a useful component of a ROS graph, both Isaac ROS Triton and TensorRT inference nodes will require application-specific `pre-processor` (`encoder`) and `post-processor` (`decoder`) nodes to handle type conversion and other necessary steps.
3030

31-
A `pre-processor` node should take in a ROS2 message, perform the pre-processing steps dictated by the model, and then convert the data into an Isaac ROS Tensor List message. For example, a `pre-processor` node could resize an image, normalize it, and then convert it into a Tensor List.
31+
A `pre-processor` node should take in a ROS 2 message, perform the pre-processing steps dictated by the model, and then convert the data into an Isaac ROS Tensor List message. For example, a `pre-processor` node could resize an image, normalize it, and then convert it into a Tensor List.
3232

33-
A `post-processor` node should be used to convert the Isaac ROS Tensor List output of the model inference into a useful ROS2 message. For example, a `post-processor` node may perform argmax to identify the class label from a classification problem.
33+
A `post-processor` node should be used to convert the Isaac ROS Tensor List output of the model inference into a useful ROS 2 message. For example, a `post-processor` node may perform argmax to identify the class label from a classification problem.
3434

3535
<div align="center">
3636

37-
![Using TensorRT or Triton](../resources/pipeline.png "Using TensorRT or Triton")
37+
![Using TensorRT or Triton](../resources/graph.png "Using TensorRT or Triton")
3838

3939
</div>
4040

‎docs/troubleshooting.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,3 +21,29 @@ One cause of this issue is when the GPU being used does not have enough memory t
2121
### Solution
2222

2323
Try using the Isaac ROS TensorRT node or the Isaac ROS Triton node with the TensorRT backend instead. Otherwise, a discrete GPU with more VRAM may be required.
24+
25+
## Triton fails to create the TensorRT engine and load a model
26+
27+
### Symptom
28+
29+
```log
30+
1: [component_container_mt-1] I0331 05:56:07.479791 11359 tensorrt.cc:5591] TRITONBACKEND_ModelInitialize: detectnet (version 1)
31+
1: [component_container_mt-1] I0331 05:56:07.483989 11359 tensorrt.cc:5640] TRITONBACKEND_ModelInstanceInitialize: detectnet (GPU device 0)
32+
1: [component_container_mt-1] I0331 05:56:08.169240 11359 logging.cc:49] Loaded engine size: 21 MiB
33+
1: [component_container_mt-1] E0331 05:56:08.209208 11359 logging.cc:43] 1: [runtime.cpp::parsePlan::314] Error Code 1: Serialization (Serialization assertion plan->header.magicTag == rt::kPLAN_MAGIC_TAG failed.)
34+
1: [component_container_mt-1] I0331 05:56:08.213483 11359 tensorrt.cc:5678] TRITONBACKEND_ModelInstanceFinalize: delete instance state
35+
1: [component_container_mt-1] I0331 05:56:08.213525 11359 tensorrt.cc:5617] TRITONBACKEND_ModelFinalize: delete model state
36+
1: [component_container_mt-1] E0331 05:56:08.214059 11359 model_lifecycle.cc:596] failed to load 'detectnet' version 1: Internal: unable to create TensorRT engine
37+
1: [component_container_mt-1] ERROR: infer_trtis_server.cpp:1057 Triton: failed to load model detectnet, triton_err_str:Invalid argument, err_msg:load failed for model 'detectnet': version 1 is at UNAVAILABLE state: Internal: unable to create TensorRT engine;
38+
1: [component_container_mt-1]
39+
1: [component_container_mt-1] ERROR: infer_trtis_backend.cpp:54 failed to load model: detectnet, nvinfer error:NVDSINFER_TRITON_ERROR
40+
1: [component_container_mt-1] ERROR: infer_simple_runtime.cpp:33 failed to initialize backend while ensuring model:detectnet ready, nvinfer error:NVDSINFER_TRITON_ERROR
41+
1: [component_container_mt-1] ERROR: Error in createNNBackend() <infer_simple_context.cpp:76> [UID = 16]: failed to initialize triton simple runtime for model:detectnet, nvinfer error:NVDSINFER_TRITON_ERROR
42+
1: [component_container_mt-1] ERROR: Error in initialize() <infer_base_context.cpp:79> [UID = 16]: create nn-backend failed, check config file settings, nvinfer error:NVDSINFER_TRITON_ERROR
43+
```
44+
45+
### Solution
46+
47+
This error can occur when TensorRT attempts to load an incompatible `model.plan` file. The incompatibility may arise due to a versioning or platform mismatch between the time of plan generation and the time of plan execution.
48+
49+
Delete the `model.plan` file that is being passed in as an argument to the Triton node's `model_repository_paths` parameter, and then use the source package's instructions to regenerate the `model.plan` file from the original weights file (often a `.etlt` or `.onnx` file).

‎isaac_ros_dnn_encoders/CMakeLists.txt‎

Lines changed: 3 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -15,56 +15,21 @@
1515
#
1616
# SPDX-License-Identifier: Apache-2.0
1717

18-
cmake_minimum_required(VERSION 3.8)
18+
cmake_minimum_required(VERSION 3.23.2)
1919
project(isaac_ros_dnn_encoders LANGUAGES C CXX)
2020

21-
# Default to C++17
22-
if(NOT CMAKE_CXX_STANDARD)
23-
set(CMAKE_CXX_STANDARD 17)
24-
endif()
25-
2621
if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
2722
add_compile_options(-Wall -Wextra -Wpedantic)
2823
endif()
2924

30-
# Default to Release build
31-
if(NOT CMAKE_BUILD_TYPE OR CMAKE_BUILD_TYPE STREQUAL "")
32-
set(CMAKE_BUILD_TYPE "Release" CACHE STRING "" FORCE)
33-
endif()
34-
message( STATUS "CMAKE_BUILD_TYPE: ${CMAKE_BUILD_TYPE}" )
35-
36-
execute_process(COMMAND uname -m COMMAND tr -d '\n'
37-
OUTPUT_VARIABLE ARCHITECTURE
38-
)
39-
message( STATUS "Architecture: ${ARCHITECTURE}" )
40-
4125
find_package(ament_cmake_auto REQUIRED)
4226
ament_auto_find_build_dependencies()
4327

44-
# Find VPI dependency
45-
find_package(vpi REQUIRED)
46-
47-
# DNN Image Encoder Node
28+
# DnnImageEncoderNode
4829
ament_auto_add_library(dnn_image_encoder_node SHARED src/dnn_image_encoder_node.cpp)
49-
target_compile_definitions(dnn_image_encoder_node
50-
PRIVATE "COMPOSITION_BUILDING_DLL"
51-
)
52-
target_link_libraries(dnn_image_encoder_node)
5330
rclcpp_components_register_nodes(dnn_image_encoder_node "nvidia::isaac_ros::dnn_inference::DnnImageEncoderNode")
5431
set(node_plugins "${node_plugins}nvidia::isaac_ros::dnn_inference::DnnImageEncoderNode;$<TARGET_FILE:dnn_image_encoder_node>\n")
5532

56-
# Install config directory
57-
install(
58-
DIRECTORY config
59-
DESTINATION share/${PROJECT_NAME}
60-
)
61-
62-
install(TARGETS dnn_image_encoder_node
63-
ARCHIVE DESTINATION lib
64-
LIBRARY DESTINATION lib
65-
RUNTIME DESTINATION bin
66-
)
67-
6833
if(BUILD_TESTING)
6934
find_package(ament_lint_auto REQUIRED)
7035
ament_lint_auto_find_test_dependencies()
@@ -73,7 +38,6 @@ if(BUILD_TESTING)
7338
add_launch_test(test/isaac_ros_dnn_image_encoder_test.py)
7439
add_launch_test(test/isaac_ros_dnn_image_encoder_image_norm_test.py)
7540
add_launch_test(test/isaac_ros_dnn_image_encoder_image_resize_test.py)
76-
7741
endif()
7842

79-
ament_auto_package(INSTALL_TO_SHARE)
43+
ament_auto_package(INSTALL_TO_SHARE config)

‎isaac_ros_dnn_encoders/include/isaac_ros_dnn_encoders/dnn_image_encoder_node.hpp‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,13 @@ namespace isaac_ros
3131
namespace dnn_inference
3232
{
3333

34+
enum class ResizeMode
35+
{
36+
kDistort = 0,
37+
kPad = 1,
38+
kCrop = 2
39+
};
40+
3441
class DnnImageEncoderNode : public nitros::NitrosNode
3542
{
3643
public:
@@ -47,6 +54,8 @@ class DnnImageEncoderNode : public nitros::NitrosNode
4754
const uint16_t network_image_height_;
4855
const std::vector<double> image_mean_;
4956
const std::vector<double> image_stddev_;
57+
int64_t num_blocks_;
58+
const ResizeMode resize_mode_;
5059
};
5160

5261
} // namespace dnn_inference

‎isaac_ros_dnn_encoders/package.xml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,21 +21,24 @@ SPDX-License-Identifier: Apache-2.0
2121
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
2222
<package format="3">
2323
<name>isaac_ros_dnn_encoders</name>
24-
<version>0.20.0</version>
24+
<version>0.30.0</version>
2525
<description>Encoders for preprocessing before running deep learning inference</description>
2626
<maintainer email="[email protected]">Hemal Shah</maintainer>
2727
<license>Apache-2.0</license>
2828
<url type="website">https://developer.nvidia.com/isaac-ros-gems/</url>
2929
<author>Ethan Yu</author>
3030
<author>Kajanan Chinniah</author>
3131
<author>Swapnesh Wani</author>
32-
32+
3333
<depend>rclcpp</depend>
3434
<depend>rclcpp_components</depend>
35+
<depend>isaac_ros_image_proc</depend>
3536
<depend>isaac_ros_nitros</depend>
3637
<depend>isaac_ros_nitros_image_type</depend>
3738
<depend>isaac_ros_nitros_tensor_list_type</depend>
3839

40+
<build_depend>isaac_ros_common</build_depend>
41+
3942
<test_depend>ament_lint_auto</test_depend>
4043
<test_depend>ament_lint_common</test_depend>
4144
<test_depend>isaac_ros_test</test_depend>

‎isaac_ros_dnn_encoders/src/dnn_image_encoder_node.cpp‎

Lines changed: 26 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -59,11 +59,11 @@ constexpr char APP_YAML_FILENAME[] = "config/dnn_image_encoder_node.yaml";
5959
constexpr char PACKAGE_NAME[] = "isaac_ros_dnn_encoders";
6060

6161
const std::vector<std::pair<std::string, std::string>> EXTENSIONS = {
62-
{"isaac_ros_nitros", "gxf/std/libgxf_std.so"},
63-
{"isaac_ros_nitros", "gxf/cuda/libgxf_cuda.so"},
64-
{"isaac_ros_nitros", "gxf/serialization/libgxf_serialization.so"},
65-
{"isaac_ros_nitros", "gxf/tensorops/libgxf_tensorops.so"},
66-
{"isaac_ros_nitros", "gxf/libgxf_message_compositor.so"}
62+
{"isaac_ros_gxf", "gxf/lib/std/libgxf_std.so"},
63+
{"isaac_ros_gxf", "gxf/lib/cuda/libgxf_cuda.so"},
64+
{"isaac_ros_gxf", "gxf/lib/serialization/libgxf_serialization.so"},
65+
{"isaac_ros_image_proc", "gxf/lib/image_proc/libgxf_tensorops.so"},
66+
{"isaac_ros_gxf", "gxf/lib/libgxf_message_compositor.so"}
6767
};
6868
const std::vector<std::string> PRESET_EXTENSION_SPEC_NAMES = {
6969
"isaac_ros_dnn_encoders",
@@ -108,7 +108,10 @@ DnnImageEncoderNode::DnnImageEncoderNode(const rclcpp::NodeOptions options)
108108
network_image_width_(declare_parameter<uint16_t>("network_image_width", 0)),
109109
network_image_height_(declare_parameter<uint16_t>("network_image_height", 0)),
110110
image_mean_(declare_parameter<std::vector<double>>("image_mean", {0.5, 0.5, 0.5})),
111-
image_stddev_(declare_parameter<std::vector<double>>("image_stddev", {0.5, 0.5, 0.5}))
111+
image_stddev_(declare_parameter<std::vector<double>>("image_stddev", {0.5, 0.5, 0.5})),
112+
num_blocks_(declare_parameter<int64_t>("num_blocks", 40)),
113+
resize_mode_(static_cast<ResizeMode>(
114+
declare_parameter<int>("resize_mode", static_cast<int>(ResizeMode::kDistort))))
112115
{
113116
if (network_image_width_ == 0) {
114117
throw std::invalid_argument(
@@ -168,6 +171,10 @@ void DnnImageEncoderNode::postLoadGraphCallback()
168171
getNitrosContext().setParameterUInt64(
169172
"resizer", "nvidia::cvcore::tensor_ops::Resize", "output_height", network_image_height_);
170173

174+
getNitrosContext().setParameterBool(
175+
"resizer", "nvidia::cvcore::tensor_ops::Resize", "keep_aspect_ratio",
176+
resize_mode_ != ResizeMode::kDistort);
177+
171178
const gxf::optimizer::ComponentInfo output_comp_info = {
172179
"nvidia::gxf::Vault", // component_type_name
173180
"vault", // component_name
@@ -199,6 +206,19 @@ void DnnImageEncoderNode::postLoadGraphCallback()
199206
getNitrosContext().setParameterUInt64(
200207
"reshaper", "nvidia::gxf::BlockMemoryPool", "block_size", block_size * sizeof(float));
201208

209+
// The minimum number of memory blocks is set based on the receiver queue capacity
210+
uint64_t num_blocks = std::max(static_cast<int>(num_blocks_), 40);
211+
getNitrosContext().setParameterUInt64(
212+
"resizer", "nvidia::gxf::BlockMemoryPool", "num_blocks", num_blocks);
213+
getNitrosContext().setParameterUInt64(
214+
"color_space_converter", "nvidia::gxf::BlockMemoryPool", "num_blocks", num_blocks);
215+
getNitrosContext().setParameterUInt64(
216+
"normalizer", "nvidia::gxf::BlockMemoryPool", "num_blocks", num_blocks);
217+
getNitrosContext().setParameterUInt64(
218+
"interleaved_to_planar", "nvidia::gxf::BlockMemoryPool", "num_blocks", num_blocks);
219+
getNitrosContext().setParameterUInt64(
220+
"reshaper", "nvidia::gxf::BlockMemoryPool", "num_blocks", num_blocks);
221+
202222
std::vector<int32_t> final_tensor_shape{1,
203223
static_cast<int32_t>(image_type_to_channel_size.at(image_type->second)),
204224
static_cast<int32_t>(network_image_height_),

‎isaac_ros_dnn_encoders/test/isaac_ros_dnn_image_encoder_image_resize_test.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,8 @@ def generate_test_description():
5454
'network_image_width': NETWORK_IMAGE_WIDTH,
5555
'network_image_height': NETWORK_IMAGE_HEIGHT,
5656
'image_mean': list(IMAGE_MEAN),
57-
'image_stddev': list(IMAGE_STDDEV)
57+
'image_stddev': list(IMAGE_STDDEV),
58+
'resize_mode': 1 # Pad mode
5859
}],
5960
remappings=[('encoded_tensor', 'tensors')])
6061

‎isaac_ros_dnn_inference_test/CMakeLists.txt‎

Lines changed: 3 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -15,58 +15,28 @@
1515
#
1616
# SPDX-License-Identifier: Apache-2.0
1717

18-
cmake_minimum_required(VERSION 3.5)
18+
cmake_minimum_required(VERSION 3.23.2)
1919
project(isaac_ros_dnn_inference_test LANGUAGES C CXX)
2020

21-
# Default to C++17
22-
if(NOT CMAKE_CXX_STANDARD)
23-
set(CMAKE_CXX_STANDARD 17)
24-
endif()
25-
2621
if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
2722
add_compile_options(-Wall -Wextra -Wpedantic)
2823
endif()
2924

30-
# Default to Release build
31-
if(NOT CMAKE_BUILD_TYPE OR CMAKE_BUILD_TYPE STREQUAL "")
32-
set(CMAKE_BUILD_TYPE "Release" CACHE STRING "" FORCE)
33-
endif()
34-
message( STATUS "CMAKE_BUILD_TYPE: ${CMAKE_BUILD_TYPE}" )
35-
36-
execute_process(COMMAND uname -m COMMAND tr -d '\n'
37-
OUTPUT_VARIABLE ARCHITECTURE
38-
)
39-
message( STATUS "Architecture: ${ARCHITECTURE}" )
40-
4125
find_package(ament_cmake_auto REQUIRED)
4226
ament_auto_find_build_dependencies()
4327

4428
# test_tensor_publisher_node
4529
ament_auto_add_library(test_tensor_publisher_node SHARED src/test_tensor_publisher_node.cpp)
46-
target_compile_definitions(test_tensor_publisher_node
47-
PRIVATE "COMPOSITION_BUILDING_DLL"
48-
)
49-
target_link_libraries(test_tensor_publisher_node)
5030
rclcpp_components_register_nodes(test_tensor_publisher_node "nvidia::isaac_ros::dnn_inference::TestTensorPublisherNode")
5131
set(node_plugins "${node_plugins}nvidia::isaac_ros::dnn_inference::TestTensorPublisherNode;$<TARGET_FILE:test_tensor_publisher_node>\n")
5232

5333
# run test tensor publisher executable
54-
ament_auto_add_executable("run_test_publisher"
55-
src/test_tensor_publisher_main.cpp
56-
)
57-
58-
target_link_libraries("run_test_publisher" test_tensor_publisher_node)
59-
60-
install(TARGETS "run_test_publisher"
61-
ARCHIVE DESTINATION lib
62-
LIBRARY DESTINATION lib
63-
RUNTIME DESTINATION bin
64-
)
34+
ament_auto_add_executable(run_test_publisher src/test_tensor_publisher_main.cpp)
35+
target_link_libraries(run_test_publisher test_tensor_publisher_node)
6536

6637
if(BUILD_TESTING)
6738
find_package(ament_lint_auto REQUIRED)
6839
ament_lint_auto_find_test_dependencies()
69-
7040
endif()
7141

7242
ament_auto_package()

0 commit comments

Comments
 (0)