Skip to content

Commit 7bd3312

Browse files
hemalshahNVhemalshahNV
authored
Merge pull request #23 from NVIDIA-ISAAC-ROS/release-3.0
Isaac ROS 3.0.0
2 parents 5f458ea + 7ee6224 commit 7bd3312

38 files changed

+811
-583
lines changed

‎README.md‎

Lines changed: 20 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,8 @@ The graph under benchmark is measured multiple times, with calculated results in
3333
- [Supported Platforms](#supported-platforms)
3434
- [Quickstart](#quickstart)
3535
- [Datasets](#datasets)
36+
- [r2b Dataset 2024](#r2b-dataset-2024)
37+
- [r2b Dataset 2023](#r2b-dataset-2023)
3638
- [Results](#results)
3739
- [Example Results](#example-results)
3840
- [Explanation of the Results JSON Format](#explanation-of-the-results-json-format)
@@ -137,26 +139,38 @@ Input data for benchmarking is provided in a rosbag.
137139

138140
To provide consistency of results, we have provided multiple dataset sequences in rosbag for use with `ros2_benchmark`; input data in other rosbag(s) can be used. These dataset sequences were captured on a robot, using very high precision time synchronization between sensors. Captured sensor data includes [HAWK (2mp RGB stereo camera with IMU)](https://www.leopardimaging.com/li-ar0234cs-stereo-gmsl2-hawk/), [D455](https://www.intelrealsense.com/depth-camera-d455/) and [XT32](https://www.hesaitech.com/product/xt32/).
139141

140-
These datasets are explicitly **not** provided inside this repository. Instead, visit NGC to download the dataset [here](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2023).
142+
These datasets are explicitly **not** provided inside this repository. Instead, visit NGC to download the dataset [here](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/collections/r2bdataset).
141143

142-
You can also download the dataset with command-line tools as follows by first installing the NGC CLI.
144+
You can also download the datasets with command-line tools as follows by first installing the NGC CLI.
143145

144146
```bash
145147
wget --content-disposition https://ngc.nvidia.com/downloads/ngccli_linux.zip && unzip ngccli_linux.zip && chmod u+x ngc-cli/ngc
146148
```
147149

148-
With the NGC CLI available, you can download the dataset with the following commands:
150+
With the NGC CLI available, you can download the datasets with the following commands:
149151

150152
```bash
151-
./ngc-cli/ngc registry resource download-version "nvidia/isaac/r2bdataset2023:2"
153+
./ngc-cli/ngc registry resource download-version "nvidia/isaac/r2bdataset2023:3"
154+
./ngc-cli/ngc registry resource download-version "nvidia/isaac/r2bdataset2024:1"
152155
```
153156

154157
Then, move the datasets to their required location:
155158

156159
```bash
157-
mv r2bdataset2023_v2 assets/datasets/r2b_dataset
160+
mkdir assets/datasets/r2b_dataset
161+
mv r2bdataset2023_v3/* assets/datasets/r2b_dataset
162+
mv r2bdataset2024_v1/* assets/datasets/r2b_dataset
158163
```
159164

165+
### r2b Dataset 2024
166+
| Sequence | Size | Visual | Contents | Description |
167+
| -------------------------------------------------------------------------------------------------- | ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
168+
| [r2b_galileo](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2024) | 471M | <img alt="galileo sequence" src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ngc/r2bdataset/r2b_galileo_sequence.gif" width="300px"/> | 4x LI HAWK stereo <br> L+R 1920x1200 RGB 30fps<br><br> IMU <br> Wheel odometry | Room sequence including robots, shelving, boxes, pallets, with moving obstacles. |
169+
| [r2b_galileo2](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2024) | 986M | <img alt="galileo2 sequence" src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ngc/r2bdataset/r2b_galileo_sequence.gif" width="300px"/> | RealSense D455 <br> L+R 1280x720 IR Mono 30fps <br> Depth 1280x780 30fps <br> 1280x800 RGB 30fps <br><br> IMU <br> Wheel odometry | Room sequence including robots, shelving, boxes, pallets, with moving obstacles. |
170+
| [r2b_robotarm](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2024) | 1.4G | <img alt="robotarm sequence" src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ngc/r2bdataset/r2b_robotarm_sequence.gif" width="300px"/> | RealSense D455 <br> L+R 1280x720 IR Mono 30fps <br> Depth 1280x780 30fps <br> 1280x800 RGB 30fps <br><br> Robot joint states | Robot arm sequence containing objects on a tray. |
171+
| [r2b_whitetunnel](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2024) | 503M | <img alt="whitetunnel sequence" src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ngc/r2bdataset/r2b_whitetunnel_sequence.gif" width="300px"/> | 4x LI HAWK stereo <br> L+R 1920x1200 RGB 30fps<br><br> IMU <br> Wheel odometry | Hallway sequence with low to no feature not-perpendicular walls, specular highlights, and reflections. |
172+
173+
### r2b Dataset 2023
160174
| Sequence | Size | Visual | Contents | Description |
161175
| ------------------------------------------------------------------------------------------------- | ---- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
162176
| [r2b_lounge](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/resources/r2bdataset2023) | 3.9G | <img alt="lounge sequence" src="resources/r2b_lounge_sequence.gif" width="300px"/> | LI HAWK stereo <br> L+R 1920x1200 RGB 30fps<br><br> RealSense D455 <br> L+R 1280x720 Mono IR 30fps <br> Depth 1280x780 30fps <br> 1280x800 RGB 30fps <br> Hesai XT32 10Hz <br> | Lounge sequence containing couch, table, chairs, and staircase with natural planted background wall. |
@@ -413,5 +427,6 @@ Besides benchmarking a graph with data loaded from rosbags, `ros2_benchmark` als
413427

414428
| Date | Changes |
415429
| ---------- | ----------------------------------- |
430+
| 2024-05-30 | Include r2b Dataset 2024 |
416431
| 2023-11-16 | Add support for live benchmark mode |
417432
| 2023-04-05 | Initial release |

‎ros2_benchmark/CMakeLists.txt‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
2-
# Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# Copyright (c) 2023-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
#
44
# Licensed under the Apache License, Version 2.0 (the "License");
55
# you may not use this file except in compliance with the License.
@@ -94,9 +94,9 @@ if(BUILD_TESTING)
9494
endif()
9595

9696
find_package(launch_testing_ament_cmake REQUIRED)
97-
add_launch_test(test/data_loader_node_pol.py TIMEOUT "15")
98-
add_launch_test(test/monitor_node_pol.py TIMEOUT "15")
99-
add_launch_test(test/playback_node_pol.py TIMEOUT "15")
97+
add_launch_test(test/data_loader_node_pol.py TIMEOUT "60")
98+
add_launch_test(test/monitor_node_pol.py TIMEOUT "60")
99+
add_launch_test(test/playback_node_pol.py TIMEOUT "60")
100100
endif()
101101

102-
ament_auto_package()
102+
ament_auto_package(INSTALL_TO_SHARE ../scripts)

‎ros2_benchmark/include/ros2_benchmark/common.hpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
// SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
2-
// Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
// Copyright (c) 2023-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
//
44
// Licensed under the Apache License, Version 2.0 (the "License");
55
// you may not use this file except in compliance with the License.
66
// You may obtain a copy of the License at
77
//
8-
// http://www.apache.org/licenses/LICENSE-2.0
8+
// http://www.apache.org/licenses/LICENSE-2.0
99
//
1010
// Unless required by applicable law or agreed to in writing, software
1111
// distributed under the License is distributed on an "AS IS" BASIS,

‎ros2_benchmark/include/ros2_benchmark/data_loader_node.hpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
// SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
2-
// Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
// Copyright (c) 2023-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
//
44
// Licensed under the Apache License, Version 2.0 (the "License");
55
// you may not use this file except in compliance with the License.
66
// You may obtain a copy of the License at
77
//
8-
// http://www.apache.org/licenses/LICENSE-2.0
8+
// http://www.apache.org/licenses/LICENSE-2.0
99
//
1010
// Unless required by applicable law or agreed to in writing, software
1111
// distributed under the License is distributed on an "AS IS" BASIS,

‎ros2_benchmark/include/ros2_benchmark/monitor_node.hpp‎

Lines changed: 21 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
// SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
2-
// Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
// Copyright (c) 2023-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
//
44
// Licensed under the Apache License, Version 2.0 (the "License");
55
// you may not use this file except in compliance with the License.
66
// You may obtain a copy of the License at
77
//
8-
// http://www.apache.org/licenses/LICENSE-2.0
8+
// http://www.apache.org/licenses/LICENSE-2.0
99
//
1010
// Unless required by applicable law or agreed to in writing, software
1111
// distributed under the License is distributed on an "AS IS" BASIS,
@@ -18,24 +18,26 @@
1818
#ifndef ROS2_BENCHMARK__MONITOR_NODE_HPP_
1919
#define ROS2_BENCHMARK__MONITOR_NODE_HPP_
2020

21-
#include <map>
2221
#include <memory>
2322
#include <string>
23+
#include <utility>
24+
#include <vector>
2425

2526
#include "common.hpp"
2627

2728
#include "rclcpp/rclcpp.hpp"
2829
#include "rclcpp/serialization.hpp"
2930

3031
#include "ros2_benchmark_interfaces/srv/start_monitoring.hpp"
32+
#include "ros2_benchmark_interfaces/srv/stop_monitoring.hpp"
3133

3234
namespace ros2_benchmark
3335
{
3436

3537
using TimePt = std::chrono::time_point<std::chrono::system_clock>;
36-
using KeyTimePtMap = std::map<int32_t, TimePt>;
38+
using KeyTimePtPairVector = std::vector<std::pair<int32_t, TimePt>>;
3739

38-
constexpr char kMonitorNodeServiceBaseName[] = "start_monitoring";
40+
constexpr char kMonitorNodeServiceBaseName[] = "monitor_node";
3941

4042
class MonitorNode : public rclcpp::Node
4143
{
@@ -57,8 +59,13 @@ class MonitorNode : public rclcpp::Node
5759

5860
/// Callback function to start monitoring the incoming messages.
5961
void StartMonitoringServiceCallback(
60-
const ros2_benchmark_interfaces::srv::StartMonitoring::Request::SharedPtr,
61-
ros2_benchmark_interfaces::srv::StartMonitoring::Response::SharedPtr response);
62+
const ros2_benchmark_interfaces::srv::StartMonitoring::Request::SharedPtr request,
63+
ros2_benchmark_interfaces::srv::StartMonitoring::Response::SharedPtr);
64+
65+
/// Callback function to stop monitoring the incoming messages.
66+
void StopMonitoringServiceCallback(
67+
const ros2_benchmark_interfaces::srv::StopMonitoring::Request::SharedPtr,
68+
ros2_benchmark_interfaces::srv::StopMonitoring::Response::SharedPtr response);
6269

6370
/// Record a satrt timestamp for a given message key.
6471
bool RecordStartTimestamp(
@@ -96,20 +103,24 @@ class MonitorNode : public rclcpp::Node
96103
std::shared_ptr<rclcpp::SubscriptionBase> monitor_sub_{nullptr};
97104

98105
std::mutex is_monitoring_mutex_;
99-
bool is_monitoring_;
106+
bool is_monitoring_{false};
100107

101108
/// A list for storing start timestamps obtained from the observed messages.
102-
KeyTimePtMap start_timestamps_{};
109+
KeyTimePtPairVector start_timestamps_{};
103110

104111
/// A list for storing timestamps of the observed messages.
105-
KeyTimePtMap end_timestamps_{};
112+
KeyTimePtPairVector end_timestamps_{};
106113

107114
/// Callback group for services.
108115
const rclcpp::CallbackGroup::SharedPtr service_callback_group_;
109116

110117
/// A service object for StartMonitoring.
111118
rclcpp::Service<ros2_benchmark_interfaces::srv::StartMonitoring>::SharedPtr
112119
start_monitoring_service_;
120+
121+
/// A service object for StopMonitoring.
122+
rclcpp::Service<ros2_benchmark_interfaces::srv::StopMonitoring>::SharedPtr
123+
stop_monitoring_service_;
113124
};
114125

115126
} // namespace ros2_benchmark

‎ros2_benchmark/include/ros2_benchmark/playback_node.hpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
// SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
2-
// Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
// Copyright (c) 2023-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
//
44
// Licensed under the Apache License, Version 2.0 (the "License");
55
// you may not use this file except in compliance with the License.
66
// You may obtain a copy of the License at
77
//
8-
// http://www.apache.org/licenses/LICENSE-2.0
8+
// http://www.apache.org/licenses/LICENSE-2.0
99
//
1010
// Unless required by applicable law or agreed to in writing, software
1111
// distributed under the License is distributed on an "AS IS" BASIS,

‎ros2_benchmark/package.xml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,10 @@ SPDX-License-Identifier: Apache-2.0
2121
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
2222
<package format="3">
2323
<name>ros2_benchmark</name>
24-
<version>2.1.0</version>
24+
<version>3.0.0</version>
2525
<description>Benchmarking framework for ROS 2 graphs</description>
2626

27-
<maintainer email="hemals@nvidia.com">Hemal Shah</maintainer>
27+
<maintainer email="isaac-ros-maintainers@nvidia.com">Isaac ROS Maintainers</maintainer>
2828
<license>Apache-2.0</license>
2929
<url type="website">https://developer.nvidia.com/isaac-ros-gems/</url>
3030
<author>CY Chen</author>

‎ros2_benchmark/ros2_benchmark/__init__.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
2-
# Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# Copyright (c) 2023-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
#
44
# Licensed under the Apache License, Version 2.0 (the "License");
55
# you may not use this file except in compliance with the License.
@@ -22,6 +22,7 @@
2222
from .ros2_benchmark_config import ROS2BenchmarkConfig
2323
from .ros2_benchmark_test import BenchmarkMetadata, ROS2BenchmarkTest
2424
from .utils.image_utility import ImageResolution, Resolution
25+
from .utils.resource_metrics import ResourceMetrics
2526

2627
__all__ = [
2728
'BasicPerformanceCalculator',
@@ -31,6 +32,7 @@
3132
'ImageResolution',
3233
'MonitorPerformanceCalculatorsInfo',
3334
'Resolution',
35+
'ResourceMetrics',
3436
'ROS2BenchmarkConfig',
3537
'ROS2BenchmarkTest',
3638
]

‎ros2_benchmark/ros2_benchmark/basic_performance_calculator.py‎

Lines changed: 10 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
2-
# Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# Copyright (c) 2023-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
#
44
# Licensed under the Apache License, Version 2.0 (the "License");
55
# you may not use this file except in compliance with the License.
@@ -81,8 +81,6 @@ def calculate_performance(self,
8181
end_timestamps_ns: dict) -> dict:
8282
"""Calculate performance based on message start and end timestamps."""
8383
perf_data = {}
84-
num_of_frame_sent = len(start_timestamps_ns)
85-
num_of_frame_dropped = len(start_timestamps_ns) - len(end_timestamps_ns)
8684

8785
# BasicPerformanceMetrics.RECEIVED_DURATION
8886
last_end_timestamp_ms = list(end_timestamps_ns.values())[-1] / 10**6
@@ -97,10 +95,15 @@ def calculate_performance(self,
9795
sent_duration_ms = last_sent_time_ms - first_sent_time_ms
9896
perf_data[BasicPerformanceMetrics.MEAN_PLAYBACK_FRAME_RATE] = len(
9997
start_timestamps_ns) / (sent_duration_ms / 1000.0)
100-
else:
101-
self.get_logger().warning(
102-
'Could not compute MEAN_PLAYBACK_FRAME_RATE due to insufficient start '
103-
f'timestamps received: {len(start_timestamps_ns)} was received')
98+
99+
perf_data[BasicPerformanceMetrics.FIRST_SENT_RECEIVED_LATENCY] = \
100+
first_end_timestamp_ms - first_sent_time_ms
101+
perf_data[BasicPerformanceMetrics.LAST_SENT_RECEIVED_LATENCY] = \
102+
last_end_timestamp_ms - last_sent_time_ms
103+
104+
perf_data[BasicPerformanceMetrics.NUM_MISSED_FRAMES] = \
105+
len(start_timestamps_ns) - len(end_timestamps_ns)
106+
perf_data[BasicPerformanceMetrics.NUM_FRAMES_SENT] = len(start_timestamps_ns)
104107

105108
# BasicPerformanceMetrics.MEAN_FRAME_RATE
106109
if received_duration_ms > 0:
@@ -115,14 +118,6 @@ def calculate_performance(self,
115118
f'#start_timestamps={len(start_timestamps_ns)} '
116119
f'#end_timestamps = {len(end_timestamps_ns)}')
117120

118-
perf_data[BasicPerformanceMetrics.NUM_MISSED_FRAMES] = num_of_frame_dropped
119-
perf_data[BasicPerformanceMetrics.NUM_FRAMES_SENT] = num_of_frame_sent
120-
121-
perf_data[BasicPerformanceMetrics.FIRST_SENT_RECEIVED_LATENCY] = \
122-
first_end_timestamp_ms - first_sent_time_ms
123-
perf_data[BasicPerformanceMetrics.LAST_SENT_RECEIVED_LATENCY] = \
124-
last_end_timestamp_ms - last_sent_time_ms
125-
126121
if self._message_key_match:
127122
# Calculate latency between sent and received messages
128123
end_to_end_latencies_ms = []

0 commit comments

Comments
 (0)