UR and Robotiq hardware interface RTDE collision

[KrzysztofDubel]

Affected ROS2 Driver version(s)

humble

Used ROS distribution.

Humble

Which combination of platform is the ROS driver running on.

Ubuntu Linux with standard kernel

How is the UR ROS2 Driver installed.

Build both the ROS driver and UR Client Library from source

Which robot platform is the driver connected to.

Real robot

Robot SW / URSim version(s)

Polyscope X - 10.11.0

How is the ROS driver used.

Headless without using the teach pendant

Issue details

Summary

Configuring Robotiq Hardware Interface causes Pipeline producer overflowed! <RTDE Data Pipeline>

Issue details

I am launching a ROS2 Control Controller Manager node that starts two Hardware Interfaces (UR and Robotiq).

• The UR hardware interface starts the socat server and client and sets the tool voltage properly.
• The UR hardware interface is configured and activated properly.
• Then the Robotiq hardware interface tries sending deactivate/activate commands through the serial interface.

This is when I get flooded with the overflow errors:

Pipeline producer overflowed! <RTDE Data Pipeline>

and the RTDE with the whole reverse port connection drops.

My ur_robot (and hw interface) parameters are as follows:

        <xacro:ur_robot
            name="${name}"
            tf_prefix="${tf_prefix}"
            parent="${parent}"
            robot_ip="${robot_ip}"
            use_fake_hardware="${use_fake_hardware}"
            fake_sensor_commands="${use_fake_hardware}"
            headless_mode="true"
            initial_positions="${xacro.load_yaml(initial_positions_file)}"
            joint_limits_parameters_file="${joint_limits_file}"
            kinematics_parameters_file="${kinematics_file}"
            physical_parameters_file="$(find ur_description)/config/${robot}/physical_parameters.yaml"
            visual_parameters_file="$(find ur_description)/config/${robot}/visual_parameters.yaml"
            use_tool_communication="true"
            tool_voltage="24"
            tool_baud_rate="115200"
            tool_parity="0"
            tool_stop_bits="1"
            tool_rx_idle_chars="1.5"
            tool_tx_idle_chars="3.5"
            tool_device_name="${tool_device_name}"
            script_filename="$(find ur_client_library)/resources/external_control.urscript"
            input_recipe_filename="$(find ur_robot_driver)/resources/rtde_input_recipe.txt"
            output_recipe_filename="$(find ur_robot_driver)/resources/rtde_output_recipe.txt">
            <xacro:insert_block name="origin" />
        </xacro:ur_robot>

Steps to Reproduce

1. Create a URDF that starts the UR hardware interface and the Robotiq hardware interface (use ur_robot macro and the robotiq_gripper macro)
2. Use the proper parameters to establish the link
3. Launch the controller manager
4. Activate the UR hardware interface first
5. Activate the Robotiq hardware interface secondSwitching the order of the hardware interfaces does not work because at that point there is no COM port socat link.

Expected Behavior

Both hardware interfaces working properly.

Actual Behavior

Pipeline producer overflowed! <RTDE Data Pipeline>

and Reverse Port connection dropping.

Workaround Suggestion

The only way I have been able to solve the issue with is to:

1. Configure and Activate the UR hardware interface
2. Deactivate and Unconfigure the UR hardware interface
3. Configure and Activate the Robotiq hardware interface
4. Configure and Activate the UR hardware interface

Relevant log output

[ur_ros2_control_node-6] [INFO] [1765816332.545433707] [resource_manager]: 'configure' hardware 'manipulator' 
[ur_ros2_control_node-6] [INFO] [1765816332.545499330] [URPositionHardwareInterface]: Starting ...please wait...
[ur_ros2_control_node-6] [INFO] [1765816332.545561375] [URPositionHardwareInterface]: Initializing driver...
[ur_ros2_control_node-6] [INFO] [1765816332.546643045] [UR_Client_Library:]: SCHED_FIFO OK, priority 99
[ur_ros2_control_node-6] [INFO] [1765816332.548147317] [UR_Client_Library:]: Negotiated RTDE protocol version to 2.
[ur_ros2_control_node-6] [INFO] [1765816332.548520047] [UR_Client_Library:]: Setting up RTDE communication with frequency 500.000000
[ur_ros2_control_node-6] [INFO] [1765816333.596684573] [UR_Client_Library:]: Starting primary client pipeline
[ur_ros2_control_node-6] [INFO] [1765816333.720104004] [URPositionHardwareInterface]: Calibration checksum: 'calib_221927923814618764'.
[ur_ros2_control_node-6] [INFO] [1765816333.720141647] [URPositionHardwareInterface]: Calibration checked successfully.
[ur_ros2_control_node-6] [INFO] [1765816333.720190490] [URPositionHardwareInterface]: System successfully started!
[ur_ros2_control_node-6] [INFO] [1765816333.720205108] [resource_manager]: Successful 'configure' of hardware 'manipulator'
[ur_ros2_control_node-6] [INFO] [1765816333.720240091] [resource_manager]: 'activate' hardware 'manipulator' 
[ur_ros2_control_node-6] [INFO] [1765816333.720244563] [URPositionHardwareInterface]: Activating HW interface
[ur_ros2_control_node-6] [INFO] [1765816333.720247497] [resource_manager]: Successful 'activate' of hardware 'manipulator'
[ur_ros2_control_node-6] [INFO] [1765816333.720577863] [UR_Client_Library:]: SCHED_FIFO OK, priority 99
[ros2-7] Successfully set manipulator to state lifecycle_msgs.msg.State(id=3, label='active')
[ur_ros2_control_node-6] [INFO] [1765816333.747962165] [UR_Client_Library:]: Robot connected to reverse interface. Ready to receive control commands.
[INFO] [ros2-7]: process has finished cleanly [pid 363074]
[INFO] [ros2-8]: process started with pid [363242]
[ur_ros2_control_node-6] [INFO] [1765816334.628838694] [resource_manager]: 'configure' hardware 'tool' 
[ur_ros2_control_node-6] [INFO] [1765816334.628964913] [resource_manager]: Successful 'configure' of hardware 'tool'
[ur_ros2_control_node-6] [INFO] [1765816334.628984121] [resource_manager]: 'activate' hardware 'tool' 
[ur_ros2_control_node-6] [INFO] [1765816334.628995477] [DefaultDriver]: Deactivate...
[ur_ros2_control_node-6] [ERROR] [1765816334.755522722] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>
(...)
[ur_ros2_control_node-6] [ERROR] [1765816335.631499571] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>
[ur_ros2_control_node-6] [INFO] [1765816335.631633299] [UR_Client_Library:]: Connection to reverse interface dropped.
[ur_ros2_control_node-6] [ERROR] [1765816335.633496646] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>
(...)
[ur_ros2_control_node-6] [ERROR] [1765816336.855409299] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>
[ur_ros2_control_node-6] [INFO] [1765816336.855539967] [RobotiqGripperHardwareInterface]: Robotiq Gripper successfully activated!
[ur_ros2_control_node-6] [INFO] [1765816336.855584517] [resource_manager]: Successful 'activate' of hardware 'tool'

Accept Public visibility

• I agree to make this context public

[urfeex]

Hi there, thank you for reaching out! I've experienced the same thing when testing the Tool forwarder URCapX. From my understanding, the issue is that the robotiq driver blocks during it's activate. It calls its internal activate() here which contains a blocking loop.

That doesn't really fit ros2_control's framework where on_activate() should be real-time critical. I can see a couple of workarounds none of which I have tested:

• Start the hardware interfaces in inactive mode using the hardware_components_initial_state parameter and activate them one after another. Activating the gripper first and then activating the arm once the gripper is active might help.
• Change the order hardwares get activated. I am not sure whether that's possible, I'd have to dive into the resource manager's implementation for that.
• Separate the hardware interaces into two different controller_managers
• Use async hardware interfaces. I think, that's not backported to Humble, though.

---

Edit: I've just seen #1612 which would contradict the first point. I'll look into that.

[KrzysztofDubel]

I couldn't find a reliable way of swapping the order of activation inside the controller manager. I opted for calling its services later in my launch.
Things I've tried that did not work:

1. Starting the controller manager as is. The sequence then looks like this:

   • UR hw init
   • Robotiq hw init
   • Robotiq hw configure
   • Robotiq hw activate
   • UR hw configure
   • UR hw activate
     That causes the Robotiq hw to fail because the socat connection (and tool voltage) is not set yet.

2. Starting the controller manager with both HW's unconfigured. Then activating Robotiq HW. Then activating UR HW.
   Same problem as 1

3. Starting the controller manager with both HW's unconfigured. Then activating UR HW. Then activating Robotiq HW.

   • UR hw init
   • Robotiq hw init
   • UR hw configure
   • UR hw activate
   • Robotiq hw configure
   • Robotiq hw activate
     Here the Gripper activates but this causes the RTDE to fail on the UR client and the disconnect.

4. Starting the controller manager with both HW's unconfigured. Then configuring both HWs (both ways). Then activating both HWs (both ways).
   This caused the Cannot configure hardware interface with no activate #1612 issue with force mode. But even when I moved the code from on_activate to on_configure in the UR HW it did not help. I would either get the RTDE fail or the socat fail. I suspect it's because of the additional thread the UR is starting in on_configure and not because of the RTDE start in read().

[KrzysztofDubel]

The only thing that worked so far and that I'm using as a workaround is.

1. Launch controller manager with Robotiq HW unconfigured.
   • The UR HW activates setting voltage on tool, starting tool comm and starting socat server.
2. Launch the UR tool communication node to start the socat client.
3. Call service to unconfigure UR HW.
4. Call service to activate Robotiq HW.
   • The Robotiq HW activates connecting the tool, activating it and so on.
5. Call service to activate UR HW again.
   • The UR HW activates starting the RTDE.
6. Launch Controller Spawners

[urfeex]

Just for completeness, I'll reference the issue in the robotiq driver regarding the blocking activation: PickNikRobotics/ros2_robotiq_gripper#35

Regarding the startup: I can see that the issue is the timing of the tool setup. I would expect, however, that the tool should be switched on and configured after the first startup.

• The socat server is always started as soon as the Tool Forward URCapX is installed and activated. In the end, it's just a network socket forwarder of the tty device present on the robot.
• Tool voltage and tool communication have to be setup once using the respective functions. If use_tool_communication is set to true that will be a part of the program starting external control on the robot. To be able to play that program, you'll have to configure the UR hardware interface first. You can, however, also create a small program on the robot that just runs these two lines of script. Then, activating the robotiq HW should be possible without configuring the UR HW first.

I currently don't have access to all of the hardware needed for testing this, so this is all just theoretical :-)