Isaac ROS 2.1.0

Author: hemalshahNV

README.md

@@ -38,11 +38,12 @@ The graph under benchmark is measured multiple times, with calculated results in
     - [Explanation of the Results JSON Format](#explanation-of-the-results-json-format)
   - [Creating Custom Benchmark](#creating-custom-benchmark)
   - [Profiling](#profiling)
+  - [Live Benchmark Mode](#live-benchmark-mode)
   - [Updates](#updates)
 
 ## Latest Update
 
-Update 2023-04-05: Initial Release
+Update 2023-11-16: Add support for live benchmark mode
 
 ## Supported Platforms
 
@@ -181,21 +182,21 @@ Included in the log is information on the host system on which results were meas
 
 The following are the performance results measured with `ros2_benchmark` on `aarch64` and `x86_64` platforms, using ROS 2 Humble in March 2023. The table below also contains links to the packages for the nodes used in the benchmark and to the complete results JSON files.
 
-| Node                                                                                                                                                       | Input Size | Intel NUC Corei7 11th Gen                                                                                                                           | AGX Orin (CPU only)                                                                                                                               |
-| ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [AprilTag Node](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/scripts/apriltag_ros_apriltag_node.py)                                | 720p       | [90.8 fps](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/results/apriltag_ros_apriltag_node-nuc_4060ti.json)<br>11 ms        | [56.3 fps](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/results/apriltag_ros_apriltag_node-agx_orin.json)<br>18 ms        |
-| [Rectify Node](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/scripts/image_proc_rectify_node.py)                                    | 1080p      | [539 fps](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/results/image_proc_rectify_node-nuc_4060ti.json)<br>1.9 ms           | [185 fps](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/results/image_proc_rectify_node-agx_orin.json)<br>5.6 ms           |
-| [H.264 Encoder Node<br>I-frame Support](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/scripts/image_transport_h264_decoder_node.py) | 1080p      | [60.5 fps](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/results/image_transport_h264_decoder_node-nuc_4060ti.json)<br>19 ms | [28.0 fps](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/results/image_transport_h264_decoder_node-agx_orin.json)<br>37 ms |
-| [H.264 Encoder Node<br>P-frame Support](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/scripts/image_transport_h264_encoder_node.py) | 1080p      | [43.4 fps](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/results/image_transport_h264_encoder_node-nuc_4060ti.json)<br>24 ms | [10.2 fps](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/results/image_transport_h264_encoder_node-agx_orin.json)<br>95 ms |
-| [Stereo Disparity Node](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/scripts/stereo_image_proc_node.py)                            | 1080p      | [99.5 fps](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/results/stereo_image_proc_node-nuc_4060ti.json)<br>6.4 ms           | [66.5 fps](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/results/stereo_image_proc_node-agx_orin.json)<br>15 ms            |
+| Node                                                                                  | Input Size | Intel NUC Corei7 11th Gen                                                      | AGX Orin (CPU only)                                                          |
+| ------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- |
+| [AprilTag Node](scripts/apriltag_ros_apriltag_node.py)                                | 720p       | [90.8 fps](results/apriltag_ros_apriltag_node-nuc_4060ti.json)<br>11 ms        | [56.3 fps](results/apriltag_ros_apriltag_node-agx_orin.json)<br>18 ms        |
+| [Rectify Node](scripts/image_proc_rectify_node.py)                                    | 1080p      | [539 fps](results/image_proc_rectify_node-nuc_4060ti.json)<br>1.9 ms           | [185 fps](results/image_proc_rectify_node-agx_orin.json)<br>5.6 ms           |
+| [H.264 Encoder Node<br>I-frame Support](scripts/image_transport_h264_decoder_node.py) | 1080p      | [60.5 fps](results/image_transport_h264_decoder_node-nuc_4060ti.json)<br>19 ms | [28.0 fps](results/image_transport_h264_decoder_node-agx_orin.json)<br>37 ms |
+| [H.264 Encoder Node<br>P-frame Support](scripts/image_transport_h264_encoder_node.py) | 1080p      | [43.4 fps](results/image_transport_h264_encoder_node-nuc_4060ti.json)<br>24 ms | [10.2 fps](results/image_transport_h264_encoder_node-agx_orin.json)<br>95 ms |
+| [Stereo Disparity Node](scripts/stereo_image_proc_node.py)                            | 1080p      | [99.5 fps](results/stereo_image_proc_node-nuc_4060ti.json)<br>6.4 ms           | [66.5 fps](results/stereo_image_proc_node-agx_orin.json)<br>15 ms            |
 
-| Graph                                                                                                                             | Input Size | Intel NUC Corei7 11th Gen                                                                                                                     | AGX Orin (CPU only)                                                                                                                         |
-| --------------------------------------------------------------------------------------------------------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
-| [AprilTag Graph](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/scripts/apriltag_ros_apriltag_graph.py)     | 720p       | [88.1 fps](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/results/apriltag_ros_apriltag_graph-nuc_4060ti.json)<br>12 ms | [56.3 fps](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/results/apriltag_ros_apriltag_graph-agx_orin.json)<br>22 ms |
-| [Stereo Disparity Graph](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/scripts/stereo_image_proc_graph.py) | 1080p      | [99.4 fps](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/results/stereo_image_proc_graph-nuc_4060ti.json)<br>16 ms     | [63.5 fps](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/results/stereo_image_proc_graph-agx_orin.json)<br>28 ms     |
+| Graph                                                        | Input Size | Intel NUC Corei7 11th Gen                                                | AGX Orin (CPU only)                                                    |
+| ------------------------------------------------------------ | ---------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------- |
+| [AprilTag Graph](scripts/apriltag_ros_apriltag_graph.py)     | 720p       | [88.1 fps](results/apriltag_ros_apriltag_graph-nuc_4060ti.json)<br>12 ms | [56.3 fps](results/apriltag_ros_apriltag_graph-agx_orin.json)<br>22 ms |
+| [Stereo Disparity Graph](scripts/stereo_image_proc_graph.py) | 1080p      | [99.4 fps](results/stereo_image_proc_graph-nuc_4060ti.json)<br>16 ms     | [63.5 fps](results/stereo_image_proc_graph-agx_orin.json)<br>28 ms     |
 
 
-> **Note**: All results above are using ROS 2 nodes from open source that run computation on the CPU only. For GPU-accelerated equivalent packages, see [Isaac ROS](https://gitlab-master.nvidia.com/isaac_ros).
+> **Note**: All results above are using ROS 2 nodes from open source that run computation on the CPU only. For GPU-accelerated equivalent packages, see [Isaac ROS](https://github.com/NVIDIA-ISAAC-ROS/).
 
 ### Explanation of the Results JSON Format
 
@@ -396,16 +397,21 @@ Follow these steps to ensure that everything in the template is configured corre
 5. Insert your custom nodes declared in step 1 to the composable node container.
 6. Revise/add benchmark configurations under `ROS2BenchmarkConfig` declaration based on your custom graph.
 
-The full benchmark configuration options can be found [here](https://gitlab-master.nvidia.com/isaac_ros/ros2_benchmark/-/blob/dev/ros2_benchmark/ros2_benchmark/default_ros2_benchmark_config.yaml) in the default `ros2_benchmark` configuration file.
+The full benchmark configuration options can be found [here](ros2_benchmark/ros2_benchmark/default_ros2_benchmark_config.yaml) in the default `ros2_benchmark` configuration file.
 
 ## Profiling
 
 When seeking to optimize performance, profiling is often used to gain deep insight into the call stack, and where processing time is spent in functions. [ros2_tracing](https://github.com/ros2/ros2_tracing) provides a tracing instrumentation to better understand performance on a CPU, but lacks information on GPU acceleration.
 
 [Nsight Systems](https://developer.nvidia.com/nsight-systems) provides tracing instrumentation for CPU, GPU, and other SOC (system-on-chip) hardware accelerators for both `aarch64` and `x86_64` platforms, and is freely available for download.  We use this tooling to profile our graphs of computation in ROS, to identify areas of improvement for compute optimization, and improvement of synchronization between heterogenous computing hardware. These tools allow for comparison of before and after to inspect profile differences with the benchmark tooling.
 
+## Live Benchmark Mode
+
+Besides benchmarking a graph with data loaded from rosbags, `ros2_benchmark` also supports a live benchmark mode that enables measuring performance outcome of a graph with its data source (e.g., a grph consists of a camera node). A [Realsense benchmark script](scripts/realsense2_node.py) is provided as an example to showcase the use of the live benchmark mode.
+
 ## Updates
 
-| Date       | Changes         |
-| ---------- | --------------- |
-| 2023-04-05 | Initial release |
+| Date       | Changes                             |
+| ---------- | ----------------------------------- |
+| 2023-11-16 | Add support for live benchmark mode |
+| 2023-04-05 | Initial release                     |

assets/configs/realsense.yaml

@@ -0,0 +1,50 @@
+%YAML 1.2
+# SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
+# Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# SPDX-License-Identifier: Apache-2.0
+---
+device_type: ''
+serial_no: ''
+usb_port_id: ''
+
+rgb_camera:
+  profile: '640x480x15'
+color_qos: "SENSOR_DATA"
+
+depth_module:
+  profile: '1280x720x30'
+  emitter_enabled: 0
+  emitter_on_off: false
+depth_qos: "SENSOR_DATA"
+depth_info_qos: "SYSTEM_DEFAULT"
+
+infra_qos: "SENSOR_DATA"
+
+enable_accel: false
+enable_color: false
+enable_depth: false
+enable_gyro: false
+enable_infra1: true
+enable_infra2: true
+
+pointcloud:
+  enable: false
+
+pointcloud_texture_index: 0
+pointcloud_texture_stream: RS2_STREAM_ANY
+
+enable_sync: false
+align_depth: false

ros2_benchmark/include/ros2_benchmark/monitor_node.hpp

@@ -60,16 +60,26 @@ class MonitorNode : public rclcpp::Node
     const ros2_benchmark_interfaces::srv::StartMonitoring::Request::SharedPtr,
     ros2_benchmark_interfaces::srv::StartMonitoring::Response::SharedPtr response);
 
+  /// Record a satrt timestamp for a given message key.
+  bool RecordStartTimestamp(
+    const int32_t & message_key,
+    const std::chrono::time_point<std::chrono::system_clock> & timestamp);
+
   /// Record an end timestamp for a given message key.
   bool RecordEndTimestamp(const int32_t & message_key);
 
   /// Record an end timestamp with an automatic generated key.
   bool RecordEndTimestampAutoKey();
 
+  /// Get a timestamp from a given serialized message
+  std::chrono::time_point<std::chrono::system_clock>
+  GetTimestampFromSerializedMessage(
+    std::shared_ptr<rclcpp::SerializedMessage> serialized_message_ptr);
+
   /// Index of this monitor node.
   uint32_t monitor_index_;
 
-  /// The name of the service StartMonitoring created in this nodee
+  /// The name of the service StartMonitoring created in this node
   std::string monitor_service_name_;
 
   /// Data format of the monitor subscriber.
@@ -78,10 +88,19 @@ class MonitorNode : public rclcpp::Node
   /// Treat the header.stamp.sec field in a message as a message ID.
   bool revise_timestamps_as_message_ids_{false};
 
+  /// Record timestamps in the observed messages as start timestamps
+  bool record_start_timestamps_{false};
+
   /// A subscriber that monitors the incoming messages for a specified topic
   /// and record their arrival timestamps.
   std::shared_ptr<rclcpp::SubscriptionBase> monitor_sub_{nullptr};
 
+  std::mutex is_monitoring_mutex_;
+  bool is_monitoring_;
+
+  /// A list for storing start timestamps obtained from the observed messages.
+  KeyTimePtMap start_timestamps_{};
+
   /// A list for storing timestamps of the observed messages.
   KeyTimePtMap end_timestamps_{};
 

ros2_benchmark/package.xml

@@ -21,7 +21,7 @@ SPDX-License-Identifier: Apache-2.0
 <?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
 <package format="3">
   <name>ros2_benchmark</name>
-  <version>2.0.0</version>
+  <version>2.1.0</version>
   <description>Benchmarking framework for ROS 2 graphs</description>
 
   <maintainer email="[email protected]">Hemal Shah</maintainer>

ros2_benchmark/ros2_benchmark/basic_performance_calculator.py

@@ -215,4 +215,6 @@ def conclude_performance(self) -> dict:
                 final_perf_data[metric] = 'INVALID VALUES: NO CONCLUDED METHOD ASSIGNED'
 
         self.reset()
+        if self._report_prefix != '':
+            return {self._report_prefix: final_perf_data}
         return final_perf_data

ros2_benchmark/ros2_benchmark/default_ros2_benchmark_config.yaml

@@ -36,6 +36,9 @@ ros2_benchmark_config:
     # Use timestamp's sec field in a message header as message IDs for benchmarking
     revise_timestamps_as_message_ids: false
 
+    # Collect start timestamps from the messages observed by monitors
+    collect_start_timestamps_from_monitors: false
+
     # Enable CPU profiling during benchmarking
     enable_cpu_profiler: true
 

ros2_benchmark/ros2_benchmark/ros2_benchmark_config.py

@@ -58,6 +58,7 @@ class BenchmarkMode(Enum):
     TIMELINE = 0
     LOOPING = 1
     SWEEPING = 2
+    LIVE = 3
 
 
 class ROS2BenchmarkConfig():
@@ -70,6 +71,7 @@ class ROS2BenchmarkConfig():
     # its type is not string (as env only supports string values).
     __config_type_map = {
         'revise_timestamps_as_message_ids': bool,
+        'collect_start_timestamps_from_monitors': bool,
         'enable_cpu_profiler': bool,
         'publish_tf_messages_in_set_data': bool,
         'publish_tf_static_messages_in_set_data': bool,