'Real-time' position streaming interface(s): requirements, design and implementation

[gavanderhoorn]

This is a discussion around the following item on our roadmap:

• real-time position streaming interface (skipping MotoROS2's internal motion queue)

MotoROS2 currently supports a mode in which JointTrajectory messages can be drip-fed to the controller, one JointTrajectoryPoint at a time, but this interface uses an additional queue inside MotoROS2 -- additional, as it's a queue of JointTrajectoryPoints, which get deconstructed into increments, queued, then fed to the Yaskawa motion controller, which also queues (and filters).

For quite a few applications being able to construct such "infinite" trajectories is already sufficient, but for more on-line and reactive use-cases all the queuing and filtering introduces both additional latency, jitter and reduces determinism (ie: when will my motion be executed exactly).

The latter especially complicates synchronisation of actions that are coordinated by an off-controller entity (setting IO or reading sensor data fi), and the former two are undesirable in any robot control application which must accurately track a reference signal.

While some fundamental limitations in the current controller generation(s) (ros-industrial/motoman#219) can't be solved here in MotoROS2, the goal would be to avoid introducing additional latency as much as possible -- fi by removing queuing as much as possible or by setting certain system parameters to specific values by default when a specific motion interface is active (#48).

This new interface would not necessarily have to use ROS messages/services/actions for data exchange, but MotoROS2 would have to be able to coordinate and moderate access to that interface, as we would want to seamlessly integrate it with MotoROS2 and avoid conflicts with other control modes.

We know of work in this direction by several community members -- some of which is even publicly available. In this thread we'd like to gather ideas for the addition of this new motion interface to MotoROS2 from members in the community that either have already or are planning to work on and/or with such low-latency interfaces. We'd also like to collaborate with community members to implement the new interface.

I'll @-mention some of the accounts that have expressed interest in this topic in the past (in no particular order and I'm surely forgetting/missing some people) to help get the discussion started: @tingelst, @drolleigh, @adv4ncr, @jarvisschultz, @rr-robert, @rr-tom-noble, @smurray47, @mmatl and @emielkemmm.

[gavanderhoorn]

As a starting point perhaps: what sort of functionality should the new interface support? Streaming at which rate? Modalities (ie: position, velocity, something else)? Should it absolutely use ROS messages, or would it actually be advantageous to decouple it from the ROS side of MotoROS2?

[emielkemmm]

So, from a functionality standpoint, having a position streaming interface (similar to Fanuc's stream motion, package J519) would probably be a good starting point. That is: being able to stream commanded positions at a deterministic rate (4ms, 8ms, whatever works best with the controller).

I think decoupling the streaming from ROS messages is more flexible.

[riv-dalatorre]

I agree with the above.
In terms of rate, the closer we can get to 4ms (which I undestand is the ceiling on the controller side) the better, however for our application latency is the most important. We can live with lower rates (~10ms) if it means prioritising minimal latency and determinism. It doesn't help us to be able to stream a point every 4ms if the robot doesn't act on the points for another 15-20ms as that will be the bottleneck for improving controller bandwidth.
Until now we have been using joint position streaming, but flexibility to stream velocities might be nice to experiment with at some point.

[ari-jv]

I second most of the requirements above.

• We also need deterministic low latency (4ms or 8ms) and rate (250Hz or 125Hz).
• And the option to stream absolute poses or relative poses. Absolute poses for 100% ROS control and relative poses to correct JBI trajectories with ROS.

(Similar to the what we're used to with the Kuka RSI option. For inspiration, the existing ros2_control RSI driver implement solutions to stream absolute poses at 250Hz. We have tested it. We're not sure about 4ms latency though. I mean we don't know if there is a convenient way to inject a newly-computed-pose every 4ms.)

[ted-miller]

And the option to stream absolute poses or relative poses. Absolute poses for 100% ROS control and relative poses to correct JBI trajectories with ROS.

I have received another request to have these two options.

[domire8]

Hello all, I also second the inputs above. Something similiar to FANUC Stream Motion, ABB EGM, KUKA RSI would be highly desirable;

Streaming of position commands (or relative positions) and joint states (position and velocity) at a deterministic rate, ideally decoupled from ROS (through UDP or similar) would be very desirable

[riv-dalatorre]

On the topic of synchronisation, we're very interested in the ability to synchronously stream joint commands to different movement groups, such as robot and multi-axis turntable, robot and linear rail, robot and robot, etc.
Where is motoros2 at with this functionality?
We'd be happy to help with testing and co-development for this feature.

[ted-miller]

Where is motoros2 at with this functionality?

The motion API itself doesn't have any limitations in regard to different control-groups. And MotoROS2 doesn't distinguish between a robot arm, a turntable, or a rail. When a motion command is issued, it contains data for up to 32 groups of 8 axes each. These will all be synchronous. (Though, MR2 does have a hardcoded limit of 8 group. That could potentially be increased.)

But MotoROS2 doesn't currently have any real-time streaming capability. So we need to figure out how we should expose the motion API to the end user w/o the buffering.

[gavanderhoorn]

Context of (some of) these comments: ros-industrial/motoman#586.

What @ted-miller writes is correct: this thread is for discussing the characteristics of a potential, to-be-added 'real-time' streaming motion interface for MotoROS2.

At least for regular, trajectory based, motion execution MotoROS2 currently supports only synchronised motion across all groups. Asynchronous control over different motion groups is not supported.

[riv-dpowell]

Does this functionality have any rough date in the roadmap yet?

[ted-miller]

No. We do not have a schedule for this functionality.

If you have some ideas on how you would like it to be implemented, we would welcome the input.

[riv-dpowell]

We'll have a chat internally and get back to you

[gavanderhoorn]

Note: this is work-in-progress, so the implementation may change at any time, until it gets released.

To give it some (more) visibility, see #449 for a work-in-progress PR that goes towards what is discussed in this thread.

Note: there are inherent delays in the motion pipelines of the DX200 and YRC1000 controllers which means that while the interface contributed by #449 is essentially a thin wrapper around mpExRcsIncrementMove(..), it will not be able to remove everything identified in ros-industrial/motoman#219.