[Sim2real] Help on policy transfer to the real robot

[loic-855]

Good morning,

In my current project, I am implementing an RL policy on a ur5e setup, using RL training to create a control policy and on the real setup using the official ROS2 driver modified for my robot setup, which has two ur5e arms, a gripper and a screwdriver as tools, place on both edge of a table.
I have been reading this conversation that has a similar setup and application: #3825, with a different control method (torque based).

As my first implementation on the real robot, I wanted to use 1 arm to simply reach some pose / orientation goal randomly defined by me. For achieving this, i trained the robot in IsaacLab on such task, and it is performing well. I applied some domain randomisation on the initialization pose, the goal is random over the table too. The policy is controlling the robot by computing joint positions delta, and applies it to the next iteration.

I then wrote a custom script to use the exported policy.pt from the precedent task, and feed it the necessary observations coming from ROS2, using topic such a /joint_states. I am also computing joint position delta there, and I than add then to the current joint position and send a joint position command to the robot using the /scaled_joint_trajectory_controller of UR driver.

However the policy on the robot side is not performing well. It seems that the observations of the real robot are really ot of the distribution then should be, when the policy is running on the simulator. It results in action beging either -1 or 1, almost nothing inbetween.

I know my description of the issue is quite vague, but i wanted to know if there is a key error in the workflow that I'm doing. All my research on this issue show me that there is various method of achieved the sim2real transfer, and it is a bit hard to understand what to begin with, as not all the possiblities seem equally good / easy to implement.

Thank you in advance for any help,
Bests

Loïc