[Question] Unstable/Erratic Arm Movement After Grasping in Custom Teleop Environment (so101 arm)

[jooyoung13]

Hello Isaac Lab Team and Community,

I am currently working on an imitation learning project using Isaac Lab. I am trying to adapt the "Teleoperation and Imitation Learning with Isaac Lab Mimic" tutorial to work with a custom robot, the so101 arm.

To achieve this, I have taken the following steps:

1.Added a so101.py asset file.

2.Created a new environment configuration file for the so101 arm, referencing the Franka's stack_ik_abs_env_cfg.py as a guide.

3.I have successfully implemented keyboard teleoperation up to this point.

However, I am facing a specific problem during teleoperation. When I press the grasp key ('k') to grasp a cube, the arm successfully grasps it. But immediately after grasping, the other joints of the so101 arm begin to move erratically and uncontrollably. This behavior is different from the Franka example.

To solve this, I have tried adjusting the effort limit, stiffness, and damping parameters for the arm, but none of these changes have resolved the issue.

Additionally, I checked if any actions were being published from teleop_se3_agent.py when this unintended movement occurs. I confirmed that no actions were being published from the agent, but the robot arm was still moving erratically.

Uploading 2025-11-03 17-09-38.mp4…
In the video, you can see that I only press the grasp key ('k') and provide no other input, yet the robot arm starts to move on its own.

I followed the tutorial and set disable_gravity=True, just as in the Franka example. Could this possibly be related to the problem?

I feel that I must be missing something or overlooking another important setting. Any comments, suggestions, or ideas on what I should check would be incredibly helpful.

Thank you very much for your time and assistance.

[jooyoung13]

I noticed the video didn’t upload correctly, so I’m attaching it again.

2025-11-03.17-09-38.mp4