[bug]`ros_to_dimos` is too slow for real-time ROS image/camera/TF streams

[liuquangao]

Description

We found that the generic ros_to_dimos(...) path is a major bottleneck for real-time ROS 2 sensor streams in dimos, especially for:

• sensor_msgs/Image
• sensor_msgs/CameraInfo
• tf2_msgs/TFMessage

In our Go2 EDU + ZED setup, using the generic conversion path caused:

• noticeable RGB visualization lag in Rerun
• unreliable startup behavior for camera pinhole initialization
• less robust TF behavior than direct raw ROS handling for /tf and /tf_static

After replacing the generic path with specialized manual conversions:

• Image: raw ROS + direct bgr8 -> numpy -> Image.from_numpy(...)
• CameraInfo: raw ROS + direct field copy
• TF: raw ROS + direct per-transform conversion

the system became much more responsive and the RGB camera correctly stayed attached to the robot in Rerun.

It would be great if dimos added dedicated fast paths for these common high-rate message types instead of routing them through the generic conversion path.

Steps to Reproduce

1. Set up a ROS 2 pipeline that publishes:

   • sensor_msgs/Image on a ZED RGB topic such as /zed/zed_node/rgb/color/rect/image
   • sensor_msgs/CameraInfo on /zed/zed_node/rgb/color/rect/camera_info
   • tf2_msgs/TFMessage on /tf and /tf_static

2. Bridge these topics into dimos using the generic ROS conversion/subscription path, including ros_to_dimos(...) for image and camera messages.

3. Visualize the resulting streams in rerun_bridge, especially:

   • the RGB camera view
   • the camera frustum / image plane in 3D
   • the camera attachment relative to the robot TF tree

4. Compare that behavior against a version that uses direct raw ROS handling:

   • Image: direct bgr8 -> numpy -> Image.from_numpy(...)
   • CameraInfo: direct field copy
   • TF: direct raw ROS subscription and per-transform conversion

5. Observe that the generic path shows noticeably higher latency and less robust startup behavior, while the direct conversion path is more responsive and attaches the camera correctly in Rerun.

Logs / Error Output

Commit Hash

3de12f8

Operating system

Ubuntu24.04

Robot Model

Other (specify below)

Firmware Version

No response

Additional Context

robot is Unitree go2 Edu

[leshy]

Yes this seems very viable, It depends on your CPU also, automatic message conversion path is very naive and should be optimized for expensive messages - Image, and PointCloud2.

TF messages, CameraInfo are extremely light so I wouldn't expect those to be a problem here