LeRobot SO-ARM-100 6 DOF robotic arm manipulation with Genesis simulator and Feetech motors

There are 2 main use cases

1. sim to real, where genesis controls the physical robot
2. real to sim, where the physical robot moves will refresh the robot rendering in genesis

TOC

• Acknowledgements
• Installation
• Camera
• Calibration & Validation
• Real 2 Sim
• Policies
• Tele-operation
• Examples
• Gradio apps
• LeRobot dataset EDA

Goal

The goal is to replay an episode recorded in REAL into a SIM environment.

Real	Sim Visual	Sim Collision
episode02_real.mp4	episode_02_sim_visual.mp4	episode_02_sim_collision.mp4

Tele-operation

For live simulation, the tele-operation process should rely on a scalable asynchronous process.

AsyncTeleoperationRerun.mp4

With pub/sub, workers repeatedly poll tasks from their dedicated queue, pushing their output to rerun.io database.

See Tele-operation for more details.

%%{init: {'theme': 'base', 'themeVariables': { 'background': '#ffffff', 'primaryColor': '#e1f5fe', 'secondaryColor': '#fff3e0', 'tertiaryColor': '#f3e5f5'}}}%%
flowchart LR
    subgraph main [Cron]
        Cron["Infinite Loop - 30 Hz"]
    end

    subgraph resources [Resources]
        LeaderArm[Leader Arm]
        FollowerArm[Follower Arm]
        Webcam[Webcam]
        Genesis[Genesis]
        YOLO26[YOLO26]
    end

    subgraph workers [Worker Processes]
        LeaderRead[Leader Read]
        FollowerControl[Follower Control]
        WebcamCapture[Webcam Capture]
        DetectObjects[Detect Objects]
        SimStep[Sim Step]
    end

    subgraph fifos [FIFO Queues]
        Q1([leader_read_q])
        Q2([follower_control_q])
        Q3([sim_step_q])
        Q4([webcam_capture_q])
    end

    subgraph shm ["-"]
        SHM{{Shared Memory}}
    end

    subgraph metrics [Database]
        Rerun[(Rerun.io)]
    end

    Cron -->|"empty"| Q1
    Q1 --> LeaderRead
    LeaderRead -->|"qpos[]"| Q2
    Q2 --> FollowerControl
    FollowerControl -->|"empty"| Q4
    Q4 --> WebcamCapture
    FollowerControl -->|"qpos[]"| Q3
    Q3 --> SimStep

    LeaderRead <-->|"qpos[]"| LeaderArm
    FollowerControl <-->|"qpos[]"| FollowerArm
    WebcamCapture <-->|"BGR[][]"| Webcam
    SimStep <-->|"RGB[][]"| Genesis

    WebcamCapture -->|"BGR[][]"| SHM
    SHM -->|"BGR[][]"| DetectObjects
    DetectObjects <-->|"predict"| YOLO26

    Cron --> Rerun
    LeaderRead --> Rerun
    FollowerControl --> Rerun
    WebcamCapture --> Rerun
    DetectObjects -->|"BoundingBox[]"| Rerun
    SimStep --> Rerun

Loading