Wrong acquisition timestamp carried in the ArgusCamera output entity message

[developer-mayuan]

We are running the latest isaac_ros_argus_camera (Isaac ros 2.1) with Hawk cameras on Jetson AGX Orin 64G Dev kit using Jetpack 5.1.2. We can have setup the isaac ros framework successfully using the dev container created from isaac_ros_common package. However, we found that wrong acquisition timestamp was carried in the ArgusCamera output entity message, which is always about 18 seconds later than the system steady clock.

How we observed this issue is by doing the following:

1. Clone the source code of isaac_ros_argus_camera and isaac_ros_image_pipeline package.
2. apply the following patch to isaac_ros_image_proc:

diff --git a/isaac_ros_image_proc/gxf/tensorops/extensions/tensorops/components/TensorOperator.cpp b/isaac_ros_image_proc/gxf/tensorops/extensions/tensorops/components/TensorOperator.cpp
index 572751d..8de17d7 100644
--- a/isaac_ros_image_proc/gxf/tensorops/extensions/tensorops/components/TensorOperator.cpp
+++ b/isaac_ros_image_proc/gxf/tensorops/extensions/tensorops/components/TensorOperator.cpp
@@ -166,6 +166,10 @@ gxf_result_t TensorOperator::tick() {
   error =
     detail::RerouteMessage<gxf::Timestamp>(output_message.value(), input_message.value(),
         [](gxf::Handle<gxf::Timestamp> output, gxf::Handle<gxf::Timestamp> input) {
+      auto steady_clock_now = std::chrono::steady_clock::now().time_since_epoch().count();
+      GXF_LOG_WARNING("frame acq time: %ld, frame pub time: %ld", input.get()->acqtime, input.get()->pubtime);
+      GXF_LOG_WARNING("system steady_clock now: %ld", steady_clock_now);
+      GXF_LOG_ERROR("Diff between current steady_clock and frame acq time: %ld", steady_clock_now - input.get()->acqtime);
           *output = *input;
           return GXF_SUCCESS;
         });

The purpose for this patch is comparing the current steady_clock timestamp with the acquisition time stamp carried in the camera message. And the reason why we put it here is because nvidia::isaac::tensor_ops::StreamConvertColorFormat is the direct down stream consumer of the ArgusCamera message entity, and TensorOperator class is the base class of the ConvertolorFormat and handles the ingress and egress messages.

4. Build isaac_ros_argus_camera and isaac_ros_image_pipeline package from source:

cd $ISAAC_ROS_WS \
 && colcon build \
   --packages-up-to isaac_ros_argus_camera \
   --parallel-workers 8 \
   --symlink-install \
   --install-base /workspaces/isaac_ros-dev/install \
   --merge-install \
 && source install/setup.bash

5. launch mono_camera or stereo_camera launch file and check the output:

ros2 launch isaac_ros_argus_camera isaac_ros_argus_camera_mono.launch.py

Here we can fount there is always an offset about 18 seconds between the frame acquisition time and the current system steady clock, which is huge. And also the acquisition time is 18 seconds later than the system clock, which doesn't make any sense. I'm wondering if anyone has any thoughts how to address this issue? Thanks!

[sgillen]

Hello, I suggest updating to latest jetpack and Isaac versions, there should not be an 18s delay for hawk on isaac_ros_argus_camera.

For Hawk camera I also recommend isaac_ros_nova/isaac_ros_hawk instead of isaac_ros_argus_camera: https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nova. The hawk package should be more convenient for you to use, and can leverage more accurate hardware timestamping. As part of the setup you can use nova-orin-init, which contains a preflight checker utility that can help diagnose any time sync issues with the system.