Generalizing this project to other robotic arms

[qingquluofeng]

How to generalize this to another robotic arm?

[fishbotics]

Hi @qingquluofeng , it wouldn't be too hard to add other arms one-by-one to this project. There are a lot of functions in here, so the specific work needed depends on what features you need for the other arms.

If you wanted to expand this to work programmatically for any arbitrary arm (say, given a URDF), that would be possible but considerably more work.

If you share more info about what you want to do (e.g. what arm you want to use, what features are most important), I can help you get started.

[jjanixe]

I am struggling with a similar issue. I would like to adapt motion policy network to work with a UR5 arm equipped with a Robotiq-85 gripper, and I am unsure where to begin. Could you provide some guidance on where to start?

[fishbotics]

Hi @jjanixe, that's awesome! A few things (also, I'd be happy to jump on a call to discuss if you're serious about pursuing this):

If you want to extend MPiNets to a new robot, you need to a few components:

1. Adapt the expert pipeline to a new robot
2. Adapt the training-time samplers for the new robot

Adapting the original "hybrid expert" pipeline for a new robot is probably going to be more trouble than it's worth. If you want to switch to a new robot, I'd recommend either using our global expert (AIT*) or some other global planner that exhibits nice motion (e.g. CuRobo). The reason for this is that the hybrid expert was based off of Geometric Fabrics, which is a closed-source NVIDIA project. You could potentially re-implement it, but it would be a lot of work to get right. We have a new paper coming out at CoRL called "Avoid Everything" that exhibits really good performance when trained using the Global Expert, so I'd probably just use that for simplicity, but I'm pretty sure other planners would work as an expert. We haven't released the code yet for Avoid Everything, but it'll be out soon. If you want access to a pre-release version, I'm happy to share it (send me an email if you'd like that)

Concretely, you're going to need to add a few things in this repo to adapt it to a new robot:

1. You need to add a new kinematics files for your robot (I can rearrange these files to be ready to support new robots): https://github.com/fishbotics/robofin/blob/main/robofin/kinematics/. FWIW, it's also fine to use a dynamic URDF-based FK module for your forward kinematics, but it'll be much slower during training than using a compiled FK implementation like I did in these files.
2. You will probably want to make robot constants for your new robot here: https://github.com/fishbotics/robofin/blob/main/robofin/robot_constants.py
3. I'd recommend adding the URDF to this folder: https://github.com/fishbotics/robofin/tree/main/robofin/urdf
4. You should add analogous functions for your robot here: https://github.com/fishbotics/robofin/blob/main/robofin/robots.py Note that the hardest thing to get right here will be the IK--this is because our IK for the Franka is generated using ikfast. If I were you, I would probably just skip implementing that (maybe raise an error in the corresponding function) and instead just use TracIK or something similar in your downstream expert pipeline. It won't be as fast, but it's only really used for expert generation and dealing with ikfast is notoriously challenging.
5. You'll have to add new point cloud samplers for your robot here. This should be pretty easy if you've implemented the kinematics above: https://github.com/fishbotics/robofin/blob/main/robofin/samplers.py
6. This class could be updated to be robot agnostic (it could be passed the robot type as a constructor argument): https://github.com/fishbotics/robofin/blob/main/robofin/collision.py. These collision spheres are defined in the constants file above. This is probably going to be annoying unless you already have collision spheres defined for your robot. If you don't, you could use a tool like https://curobo.org/tutorials/1_robot_configuration.html. You'd use this to define the spheres and then add the values into robofin.

[jjanixe]

@fishbotics Thank you for your quick and detailed response! I have emailed you to request the pre-release version of the code and ask a few more questions about the kinematics.