Want to Use motoros2 with RoboManipBaselines

[yuutayuuta50]

https://github.com/isri-aist/RoboManipBaselines
I would like to use motoros2 with RoboManipBaselines.
I have confirmed that during Rollout, images and joint angles are obtained step-by-step, and the robot is controlled sequentially.
It seems that a Universal Robot is used in the example.
Is it possible to control a robot in this way using motoros2?
Also, is there any plan or possibility that this will be supported in the future?

[gavanderhoorn]

It's a bit unclear to me what sort of control interfaces are used by RoboManipBaselines, but my assumption would be they'd need a real-time motion interface -- at least when interacting with a real robot.

That's currently not supported by MotoROS2.

But there is a discussion around this, see #186.

[yuutayuuta50]

Thank you for the information. For now, I’m planning to handle this by making the AI generate segmented trajectories that can be executed more easily. It would be great if in the future we could directly input AI-generated outputs into the system like this.

[gavanderhoorn]

If you could clarify what sort of control interfaces are needed / used with the other robots in RoboManipBaselines, that could help us understand what the needed changes to MotoROS2 would be.

[yuutayuuta50]

We are using models such as ACT and DiffusionPolicy as provided in the GitHub RoboManipBaselines repository. These models take the current joint angles and an image as input, and output the next target joint angles. Therefore, it is necessary to obtain an image after moving to the target joint configuration.

Currently, with motoros2, we are using an algorithm that moves the robot to the target, captures an image there, and then plans the next movement. However, this process is time-consuming and visually unappealing.

If we could obtain joint angles and images during motion, calculate the next target while still moving, and append that new target to the trajectory, we believe the robot could move much more smoothly.

Although we can test this in simulation, it is not very meaningful unless the real hardware also supports it.