Ego2Exe: Transforming Egocentric Human Videos into Executable Robot Demonstrations

Ego2Exe is a practical codebase for turning egocentric human manipulation videos into executable robot demonstrations. The current pipeline reconstructs hands, removes human arms, retargets the hand motion to an RB-Y1 robot, and composites the robot back into the egocentric video.

Latest Updates

• 13/07/2026: The project structure is re-organized.
• 09/07/2026: A beta version that isn't really a beta version has been released😂😂😂.

Installation

Clone the repository and submodules:

git clone --recurse-submodules https://github.com/Agentic-Intelligence-Lab/Ego2Exe.git
cd Ego2Exe
export PYTHONPATH="$(pwd)"

Create the main environment:

conda create -n ego2exe python=3.10
conda activate ego2exe
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117

Install the project and third-party dependencies in this environment following the README files under thirdparty/ where needed.

Note: SAM3 currently needs its own environment. Use that environment only for arm segmentation, then return to ego2exe for the rest of the pipeline.

Data and Assets

Download MuJoCo robot assets:

git clone https://github.com/google-deepmind/mujoco_menagerie.git

Download the TACO data and place one sequence in the expected layout, for example:

examples/taco/
  color.mp4
  egocentric_frame_extrinsic.npy
  egocentric_intrinsic.txt
  left_hand.pkl
  left_hand_shape.pkl
  right_hand.pkl
  right_hand_shape.pkl
  tool_*.npy
  target_*.npy

Usage

1. Reconstruct hands with HaWoR:

conda activate ego2exe
python scripts/run_hand_reconstrunction.py \
  --video examples/taco/color.mp4 \
  --output-dir outputs/taco_reconstruction

2. Replay or inspect reconstructed hands:

python scripts/replay_hawor_mujoco.py \
  --input outputs/taco_reconstruction/color_slam_world_space_res.npz \
  --taco-dir examples/taco

python scripts/joint_replay.py \
  --input outputs/taco_reconstruction/color_slam_world_space_res.npz \
  --taco-dir examples/taco

3. Segment human arms with SAM3:

conda activate sam3
python scripts/run_arm_segmentation.py \
  --video examples/taco/color.mp4 \
  --output-dir outputs/taco_color_mask

4. Remove human arms with ProPainter:

conda activate ego2exe
python scripts/run_hand_removal.py \
  --video examples/taco/color.mp4 \
  --mask-dir outputs/taco_color_mask \
  --output-dir outputs/taco_inpaint

5. Debug robot placement in MuJoCo:

python scripts/place_robot_mujoco.py --loop

6. Retarget the hand trajectory to RB-Y1 and replay it:

python scripts/retarget_taco_rby1.py \
  --data-dir examples/taco \
  --output outputs/taco_rby1_wrist_ik_traj.npz

python scripts/replay_taco_rby1_wrist_ik.py \
  --ik-npz outputs/taco_rby1_wrist_ik_traj.npz

7. Composite the robot into the arm-removed video:

python scripts/run_composite.py \
  --video examples/taco/color.mp4 \
  --inpaint-video outputs/taco_inpaint.mp4 \
  --ik-npz outputs/taco_rby1_wrist_ik_traj.npz \
  --output-dir outputs/taco_composite

Results

Hand Reconstruction	Arm Segmentation	Arm Removal	Robot Placement / Replay	Composite

Acknowledgements

We gratefully acknowledge these projects and datasets:

• HaWoR
• ProPainter
• SAM3
• Mink
• TACO-Instructions
• MuJoCo Menagerie