1- ### ROS2 Robot vs Camera Optical Coordination Systems
1+ ## ROS2 Robot Coordinate System vs Camera Optical Coordinate System
22
3- * Point Of View:
4- * Imagine we are standing behind of the camera, and looking forward.
5- * Always use this point of view when talking about coordinates, left vs right IRs, position of sensor, etc. .
3+ * Point of View:
4+ * Imagine standing behind the camera and looking forward.
5+ * Always use this point of view when discussing coordinates, left vs right IR, sensor positions , etc.
66
77![ ROS2 and Camera Coordinate System] ( ../image/application_guide/image0.png )
88
9- * ROS2 Coordinate System: (X: Forward, Y: Left , Z: Up)
9+ * ROS2 Coordinate System: (X: Forward, Y: Left, Z: Up)
1010* Camera Optical Coordinate System: (X: Right, Y: Down, Z: Forward)
11- * All data published in our wrapper topics is optical data taken directly from our camera sensors.
12- * static and dynamic TF topics publish optical CS and ROS CS to give the user the ability to move from one CS to other CS .
11+ * All data published in the wrapper topics is optical data taken directly from the camera sensors.
12+ * Static and dynamic TF topics publish optical and ROS coordinate systems so users can transform between them .
1313
1414### Using ROS2 TF Tools
1515
16- #### Viewing the TF Tree Structure
16+ #### View TF Tree Structure
1717
18- You can use the following ROS2 commands to print and visualize the TF tree published by the camera package:
18+ Use the following ROS2 commands to print and visualize the TF tree published by the camera package:
1919
2020** Print all TF relationships:**
2121
2222``` bash
2323ros2 run tf2_tools view_frames
2424```
2525
26- This command generates a ` frames.pdf ` file that displays the hierarchical relationships between all frames.
26+ This command generates a ` frames.pdf ` file showing the hierarchy between all frames.
2727
2828![ image-20251027111351870] ( ../image/application_guide/image4.png )
2929
@@ -33,9 +33,45 @@ This command generates a `frames.pdf` file that displays the hierarchical relati
3333ros2 topic echo /tf_static
3434```
3535
36- #### Visualizing TF Tree with rviz2
36+ ** View the TF transform between two specified frames: **
3737
38- You can use rviz2 to visualize the TF tree structure and relative positions of coordinate systems in real-time:
38+ Use the following command to view the transform between two specific frames:
39+
40+ ``` bash
41+ ros2 run tf2_ros tf2_echo [source_frame] [target_frame]
42+ ```
43+
44+ Example, view transform from ` camera_link ` to ` camera_depth_optical_frame ` :
45+
46+ ``` bash
47+ ros2 run tf2_ros tf2_echo camera_link camera_depth_optical_frame
48+ ```
49+
50+ The command continuously outputs the real-time transform between the two frames, including:
51+
52+ - Translation: x, y, z (meters)
53+ - Rotation (Quaternion): x, y, z, w
54+ - Rotation (RPY): roll, pitch, yaw (radians and degrees)
55+ - Transform Matrix: 4×4 matrix with rotation and translation
56+
57+ Sample output:
58+
59+ ```
60+ At time 0.0
61+ - Translation: [0.000, 0.000, 0.000]
62+ - Rotation: in Quaternion [-0.500, 0.500, -0.500, 0.500]
63+ - Rotation: in RPY (radian) [-1.571, -0.000, -1.571]
64+ - Rotation: in RPY (degree) [-90.000, -0.000, -90.000]
65+ - Matrix:
66+ 0.000 0.000 1.000 0.000
67+ -1.000 0.000 0.000 0.000
68+ 0.000 -1.000 0.000 0.000
69+ 0.000 0.000 0.000 1.000
70+ ```
71+
72+ #### Visualize TF Tree in rviz2
73+
74+ Use rviz2 to visualize the TF tree and relative frame poses in real time:
3975
4076``` bash
4177rviz2
@@ -44,100 +80,247 @@ rviz2
4480In rviz2:
4581
4682- Add the ` TF ` display plugin
47- - Configure the Fixed Frame to ` camera_link ` or ` camera_depth_optical_frame `
48- - Select the TF frames to display
83+ - Set the Fixed Frame to ` camera_link ` or ` camera_depth_optical_frame `
84+ - Select which TF frames to display
4985
5086![ image-20251027140652727] ( ../image/application_guide/image5.png )
5187
5288### Camera TF Calculation and Publishing Mechanism
5389
5490#### Core Function: ` OBCameraNode::calcAndPublishStaticTransform() `
5591
56- The camera node calculates and publishes static transformation relationships between all sensors through this function. Below is a detailed explanation of the code:
92+ The camera node uses this function to calculate and publish all static transforms between sensors.
93+
94+ ``` cpp
95+ void OBCameraNode::calcAndPublishStaticTransform () {
96+ tf2::Quaternion quaternion_optical, zero_rot;
97+ zero_rot.setRPY(0.0, 0.0, 0.0);
98+ quaternion_optical.setRPY(-M_PI / 2, 0.0, -M_PI / 2);
99+ tf2::Vector3 zero_trans(0, 0, 0);
100+ auto base_stream_profile = stream_profile_ [ base_stream_ ] ;
101+ auto device_info = device_ ->getDeviceInfo();
102+ CHECK_NOTNULL (device_info);
103+ auto pid = device_info->getPid();
104+ if (!base_stream_profile) {
105+ RCLCPP_ERROR_STREAM(logger_ , "Failed to get base stream profile");
106+ return;
107+ }
108+ CHECK_NOTNULL (base_stream_profile.get());
109+ for (const auto &item : stream_profile_ ) {
110+ auto stream_index = item.first;
57111
58- #### Quaternion Initialization and Coordinate System Transformation
112+ auto stream_profile = item.second;
113+ if (!stream_profile) {
114+ continue;
115+ }
116+ OBExtrinsic ex;
117+ try {
118+ ex = stream_profile->getExtrinsicTo(base_stream_profile);
119+ } catch (const ob::Error &e) {
120+ RCLCPP_ERROR_STREAM (logger_ , "Failed to get " << stream_name_ [ stream_index]
121+ << " extrinsic: " << e.getMessage());
122+ ex = OBExtrinsic({{1, 0, 0, 0, 1, 0, 0, 0, 1}, {0, 0, 0}});
123+ }
124+
125+ auto Q = rotationMatrixToQuaternion(ex.rot);
126+ Q = quaternion_optical * Q * quaternion_optical.inverse();
127+ tf2::Vector3 trans(ex.trans[0], ex.trans[1], ex.trans[2]);
128+ auto timestamp = node_->now();
129+ if (stream_index.first != base_stream_.first) {
130+ if (stream_index.first == OB_STREAM_IR_RIGHT && base_stream_.first == OB_STREAM_DEPTH) {
131+ trans[0] = std::abs(trans[0]); // because left and right ir calibration is error
132+ }
133+ publishStaticTF (timestamp, trans, Q, frame_id_ [ base_stream_ ] , frame_id_ [ stream_index] );
134+ }
135+ publishStaticTF(timestamp, zero_trans, quaternion_optical, frame_id_ [ stream_index] ,
136+ optical_frame_id_ [ stream_index] );
137+ RCLCPP_INFO_STREAM(logger_ , "Publishing static transform from " << stream_name_ [ stream_index]
138+ << " to "
139+ << stream_name_ [ base_stream_ ] );
140+ RCLCPP_INFO_STREAM(logger_ , "Translation " << trans[ 0] << ", " << trans[ 1] << ", " << trans[ 2] );
141+ RCLCPP_INFO_STREAM(logger_ , "Rotation " << Q.getX() << ", " << Q.getY() << ", " << Q.getZ()
142+ << ", " << Q.getW());
143+ }
144+
145+ if ((pid == FEMTO_BOLT_PID || pid == FEMTO_MEGA_PID) && enable_stream_ [ DEPTH] &&
146+ enable_stream_ [ COLOR] && enable_publish_extrinsic_ ) {
147+ // calc depth to color
148+
149+ CHECK_NOTNULL (stream_profile_ [ COLOR] );
150+ auto depth_to_color_extrinsics = base_stream_profile->getExtrinsicTo(stream_profile_ [ COLOR] );
151+ auto Q = rotationMatrixToQuaternion(depth_to_color_extrinsics.rot);
152+ Q = quaternion_optical * Q * quaternion_optical.inverse();
153+ publishStaticTF(node_ ->now(), zero_trans, Q, camera_link_frame_id_ , frame_id_ [ base_stream_ ] );
154+ } else {
155+ publishStaticTF(node_ ->now(), zero_trans, zero_rot, camera_link_frame_id_ ,
156+ frame_id_ [ base_stream_ ] );
157+ }
158+
159+ if (enable_stream_ [ DEPTH] && enable_stream_ [ COLOR] && enable_publish_extrinsic_ ) {
160+ static const char * frame_id = "depth_to_color_extrinsics";
161+ OBExtrinsic ex;
162+ try {
163+ ex = base_stream_profile->getExtrinsicTo(stream_profile_ [ COLOR] );
164+ } catch (const ob::Error &e) {
165+ RCLCPP_ERROR_STREAM(logger_ ,
166+ "Failed to get " << frame_id << " extrinsic: " << e.getMessage());
167+ ex = OBExtrinsic({{1, 0, 0, 0, 1, 0, 0, 0, 1}, {0, 0, 0}});
168+ }
169+ depth_to_other_extrinsics_ [ COLOR] = ex;
170+ auto ex_msg = obExtrinsicsToMsg(ex, frame_id);
171+ CHECK_NOTNULL(depth_to_other_extrinsics_publishers_ [ COLOR] );
172+ depth_to_other_extrinsics_publishers_ [ COLOR] ->publish(ex_msg);
173+ }
174+
175+ if (enable_stream_ [ DEPTH] && enable_stream_ [ INFRA0] && enable_publish_extrinsic_ ) {
176+ static const char * frame_id = "depth_to_ir_extrinsics";
177+ OBExtrinsic ex;
178+ try {
179+ ex = base_stream_profile->getExtrinsicTo(stream_profile_ [ INFRA0] );
180+ } catch (const ob::Error &e) {
181+ RCLCPP_ERROR_STREAM(logger_ ,
182+ "Failed to get " << frame_id << " extrinsic: " << e.getMessage());
183+ ex = OBExtrinsic({{1, 0, 0, 0, 1, 0, 0, 0, 1}, {0, 0, 0}});
184+ }
185+ depth_to_other_extrinsics_ [ INFRA0] = ex;
186+ auto ex_msg = obExtrinsicsToMsg(ex, frame_id);
187+ CHECK_NOTNULL(depth_to_other_extrinsics_publishers_ [ INFRA0] );
188+ depth_to_other_extrinsics_publishers_ [ INFRA0] ->publish(ex_msg);
189+ }
190+ if (enable_stream_ [ DEPTH] && enable_stream_ [ INFRA1] && enable_publish_extrinsic_ ) {
191+ static const char * frame_id = "depth_to_left_ir_extrinsics";
192+ OBExtrinsic ex;
193+ try {
194+ ex = base_stream_profile->getExtrinsicTo(stream_profile_ [ INFRA1] );
195+ } catch (const ob::Error &e) {
196+ RCLCPP_ERROR_STREAM(logger_ ,
197+ "Failed to get " << frame_id << " extrinsic: " << e.getMessage());
198+ ex = OBExtrinsic({{1, 0, 0, 0, 1, 0, 0, 0, 1}, {0, 0, 0}});
199+ }
200+ depth_to_other_extrinsics_ [ INFRA1] = ex;
201+ auto ex_msg = obExtrinsicsToMsg(ex, frame_id);
202+ CHECK_NOTNULL(depth_to_other_extrinsics_publishers_ [ INFRA1] );
203+ depth_to_other_extrinsics_publishers_ [ INFRA1] ->publish(ex_msg);
204+ }
205+ if (enable_stream_ [ DEPTH] && enable_stream_ [ INFRA2] && enable_publish_extrinsic_ ) {
206+ static const char * frame_id = "depth_to_right_ir_extrinsics";
207+ OBExtrinsic ex;
208+ try {
209+ ex = base_stream_profile->getExtrinsicTo(stream_profile_ [ INFRA2] );
210+ } catch (const ob::Error &e) {
211+ RCLCPP_ERROR_STREAM(logger_ ,
212+ "Failed to get " << frame_id << " extrinsic: " << e.getMessage());
213+ ex = OBExtrinsic({{1, 0, 0, 0, 1, 0, 0, 0, 1}, {0, 0, 0}});
214+ }
215+ ex.trans[ 0] = -std::abs(ex.trans[ 0] );
216+ depth_to_other_extrinsics_ [ INFRA2] = ex;
217+ auto ex_msg = obExtrinsicsToMsg(ex, frame_id);
218+ CHECK_NOTNULL(depth_to_other_extrinsics_publishers_ [ INFRA2] );
219+ depth_to_other_extrinsics_publishers_ [ INFRA2] ->publish(ex_msg);
220+ }
221+ if (enable_stream_ [ DEPTH] && enable_stream_ [ ACCEL] && enable_publish_extrinsic_ ) {
222+ static const char * frame_id = "depth_to_accel_extrinsics";
223+ OBExtrinsic ex;
224+ try {
225+ ex = base_stream_profile->getExtrinsicTo(stream_profile_ [ ACCEL] );
226+ } catch (const ob::Error &e) {
227+ RCLCPP_ERROR_STREAM(logger_ ,
228+ "Failed to get " << frame_id << " extrinsic: " << e.getMessage());
229+ ex = OBExtrinsic({{1, 0, 0, 0, 1, 0, 0, 0, 1}, {0, 0, 0}});
230+ }
231+ depth_to_other_extrinsics_ [ ACCEL] = ex;
232+ auto ex_msg = obExtrinsicsToMsg(ex, frame_id);
233+ CHECK_NOTNULL(depth_to_other_extrinsics_publishers_ [ ACCEL] );
234+ depth_to_other_extrinsics_publishers_ [ ACCEL] ->publish(ex_msg);
235+ }
236+ if (enable_stream_ [ DEPTH] && enable_stream_ [ GYRO] && enable_publish_extrinsic_ ) {
237+ static const char * frame_id = "depth_to_gyro_extrinsics";
238+ OBExtrinsic ex;
239+ try {
240+ ex = base_stream_profile->getExtrinsicTo(stream_profile_ [ GYRO] );
241+ } catch (const ob::Error &e) {
242+ RCLCPP_ERROR_STREAM(logger_ ,
243+ "Failed to get " << frame_id << " extrinsic: " << e.getMessage());
244+ ex = OBExtrinsic({{1, 0, 0, 0, 1, 0, 0, 0, 1}, {0, 0, 0}});
245+ }
246+ depth_to_other_extrinsics_ [ GYRO] = ex;
247+ auto ex_msg = obExtrinsicsToMsg(ex, frame_id);
248+ CHECK_NOTNULL(depth_to_other_extrinsics_publishers_ [ GYRO] );
249+ depth_to_other_extrinsics_publishers_ [ GYRO] ->publish(ex_msg);
250+ }
251+ if (enable_sync_output_accel_gyro_ ) {
252+ tf2::Quaternion zero_rot;
253+ zero_rot.setRPY(0.0, 0.0, 0.0);
254+ tf2::Vector3 zero_trans(0, 0, 0);
255+ publishStaticTF(node_ ->now(), zero_trans, zero_rot, optical_frame_id_ [ GYRO] ,
256+ accel_gyro_frame_id_ );
257+ }
258+ }
259+ ```
260+
261+ #### Function Breakdown
262+
263+ Detailed explanation of the code:
264+
265+ ** Quaternion Initialization and Coordinate Transform**
59266
60267``` cpp
61268tf2::Quaternion quaternion_optical, zero_rot;
62269zero_rot.setRPY(0.0 , 0.0 , 0.0 );
63270quaternion_optical.setRPY(-M_PI / 2 , 0.0 , -M_PI / 2 );
64271```
65272
66- ** Explanation:**
67-
68- - ` quaternion_optical ` : Defines the rotation transformation from the optical coordinate system to the ROS standard coordinate system (90-degree rotation)
69- - This rotation converts the camera optical coordinate system (X right, Y down, Z forward) to the ROS standard coordinate system (X forward, Y left, Z up)
273+ - ` quaternion_optical ` : Defines the rotation from optical coordinates to ROS standard (90° rotation)
274+ - Converts camera optical CS (X right, Y down, Z forward) to ROS CS (X forward, Y left, Z up)
70275
71- #### Obtaining Device Information and Base Stream
276+ ** Get Device Info and Base Stream**
72277
73278``` cpp
74279auto base_stream_profile = stream_profile_[base_stream_];
75280auto device_info = device_->getDeviceInfo ();
76- // The base stream is typically the DEPTH stream
281+ // Base stream usually DEPTH
77282```
78283
79- ** Explanation:**
80-
81- - A base stream (typically the depth stream) is selected, and all other sensor transformations are calculated relative to this base stream
284+ - Choose a base stream (usually depth); all other transforms are relative to it
82285
83- #### Iterating Through All Streams and Calculating Relative Transformations
286+ ** Iterate Streams and Compute Relative Transforms **
84287
85288``` cpp
86289for (const auto &item : stream_profile_) {
87290 auto stream_index = item.first;
88291 auto stream_profile = item.second;
89-
90- // Get the extrinsics of this stream relative to the base stream
91292 OBExtrinsic ex;
92293 ex = stream_profile->getExtrinsicTo(base_stream_profile);
93-
94- // Convert rotation matrix to quaternion
95294 auto Q = rotationMatrixToQuaternion(ex.rot);
96-
97- // Apply optical coordinate system transformation: Q_new = quaternion_optical * Q * quaternion_optical.inverse()
98295 Q = quaternion_optical * Q * quaternion_optical.inverse();
99-
100296 tf2::Vector3 trans(ex.trans[0], ex.trans[1], ex.trans[2]);
101297```
102298
103- **Explanation:**
104-
105- - `OBExtrinsic` contains the rotation matrix (`rot`) and translation vector (`trans`) between two sensors
106- - Quaternion multiplication applies the optical coordinate system transformation to each sensor's rotation relationship
107- - This transformation converts the camera's native optical coordinate system to the ROS standard coordinate system
299+ - `OBExtrinsic` holds rotation matrix (`rot`) and translation vector (`trans`)
300+ - Apply optical-to-ROS rotation via quaternion multiplication
108301
109- #### Publishing TF Transformations
302+ **Publish TF Transforms**
110303
111304```cpp
112- // Publish the transformation from sensor to base stream (in ROS coordinate system)
113305publishStaticTF(timestamp, trans, Q, frame_id_[base_stream_], frame_id_[stream_index]);
114-
115- // Publish the transformation from physical frame to its optical frame
116306publishStaticTF(timestamp, zero_trans, quaternion_optical, frame_id_[stream_index],
117307 optical_frame_id_[stream_index]);
118308```
119309
120- ** Explanation:**
310+ - First: base stream to sensor (translation + rotation)
311+ - Second: sensor frame to optical frame (pure rotation)
121312
122- - First ` publishStaticTF ` : Publishes the transformation from the base stream to the current sensor (translation + rotation)
123- - Second ` publishStaticTF ` : Publishes the transformation from physical frame to optical frame (pure rotation, no translation)
124- - ` frame_id_[stream_index] ` : Physical coordinate system frame name (e.g., ` camera_depth_frame ` )
125- - ` optical_frame_id_[stream_index] ` : Optical coordinate system frame name (e.g., ` camera_depth_optical_frame ` )
126-
127- #### Special Handling for Left and Right IR Cameras
313+ ** Special Handling for Left/Right IR**
128314
129315``` cpp
130316if (stream_index.first == OB_STREAM_IR_RIGHT && base_stream_.first == OB_STREAM_DEPTH) {
131317 trans[0] = std::abs(trans[0]);
132318}
133319```
134320
135- ** Explanation:**
136-
137- - Left and right IR cameras are symmetric about the center plane in the device coordinate system
138- - Using ` abs() ` ensures the X-axis offset is positive, maintaining geometric consistency
321+ - Ensures symmetry consistency between left/right IR cameras
139322
140- #### Publishing Extrinsics from Depth to Other Sensors
323+ ** Publish Depth-to- Other Extrinsics **
141324
142325``` cpp
143326if (enable_stream_[DEPTH] && enable_stream_[COLOR] && enable_publish_extrinsic_) {
@@ -147,7 +330,4 @@ if (enable_stream_[DEPTH] && enable_stream_[COLOR] && enable_publish_extrinsic_)
147330}
148331```
149332
150- ** Explanation:**
151-
152- - In addition to publishing transformation relationships through TF, raw extrinsic parameters are also published through custom topics
153- - This allows users to directly access the camera's intrinsic and extrinsic parameters for high-precision point cloud alignment and depth-color registration
333+ - Publishes raw extrinsics via topic for advanced alignment and registration
0 commit comments