Skip to content

Commit 63130c1

Browse files
committed
Add doc for locomanip teleop
1 parent 3f8b33e commit 63130c1

File tree

8 files changed

+315
-102
lines changed

8 files changed

+315
-102
lines changed
Lines changed: 3 additions & 0 deletions
Loading
Lines changed: 3 additions & 0 deletions
Loading

docs/pages/example_workflows/locomanipulation/index.rst

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
G1 Loco-Manipulation Box Pick and Place Task
22
============================================
33

4-
This example demonstrates the complete workflow for the **G1 loco-manipulation box pick and place task** in Isaac Lab - Arena, covering environment setup and validation, data generation, policy post-training, and closed-loop evaluation.
4+
This example demonstrates the complete workflow for the **G1 loco-manipulation box pick and place task** in Isaac Lab - Arena, covering environment setup and validation, teleoperation data collection (OpenXR with Meta Quest 3), data generation, policy post-training, and closed-loop evaluation.
55

66
.. image:: ../../../images/g1_galileo_arena_box_pnp_locomanip.gif
77
:align: center
@@ -56,7 +56,7 @@ including lower body locomotion, squatting, and bimanual manipulation.
5656
Workflow
5757
--------
5858

59-
This tutorial covers the pipeline between creating an environment, generating training data,
59+
This tutorial covers the pipeline between creating an environment, collecting teleoperation demonstrations, generating training data,
6060
fine-tuning a policy (GR00T N1.6), and evaluating the policy in closed-loop.
6161
A user can follow the whole pipeline, or can start at any intermediate step
6262
by downloading the pre-generated output of the preceding step(s), which we provide
@@ -90,16 +90,18 @@ Workflow Steps
9090
Follow the following steps to complete the workflow:
9191

9292
- :doc:`step_1_environment_setup`
93-
- :doc:`step_2_data_generation`
94-
- :doc:`step_3_policy_training`
95-
- :doc:`step_4_evaluation`
93+
- :doc:`step_2_teleoperation`
94+
- :doc:`step_3_data_generation`
95+
- :doc:`step_4_policy_training`
96+
- :doc:`step_5_evaluation`
9697

9798

9899
.. toctree::
99100
:maxdepth: 1
100101
:hidden:
101102

102103
step_1_environment_setup
103-
step_2_data_generation
104-
step_3_policy_training
105-
step_4_evaluation
104+
step_2_teleoperation
105+
step_3_data_generation
106+
step_4_policy_training
107+
step_5_evaluation

docs/pages/example_workflows/locomanipulation/step_2_data_generation.rst

Lines changed: 0 additions & 92 deletions
This file was deleted.
Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
Teleoperation Data Collection
2+
-----------------------------
3+
4+
This workflow covers collecting demonstrations for the G1 loco-manipulation task using **Meta Quest 3** supported by **NVIDIA CloudXR**.
5+
6+
This workflow requires several components to run:
7+
8+
* **NVIDIA CloudXR Runtime**: Runs in a Docker container on your workstation and streams the Isaac Lab simulation to a compatible XR device. See the `CloudXR Runtime documentation <https://docs.nvidia.com/cloudxr-sdk/latest/usr_guide/cloudxr_runtime/index.html>`_.
9+
* **Arena Docker container**: Runs the Isaac Lab simulation and recording.
10+
* **CloudXR.js WebServer**: Meta Quest 3 and Pico 4 Ultra connect to Isaac Lab via the CloudXR.js WebXR client. See `CloudXR.js (Early Access) <https://docs.nvidia.com/cloudxr-sdk/latest/usr_guide/cloudxr_js/index.html>`_.
11+
12+
.. note::
13+
14+
You must join the **NVIDIA CloudXR Early Access Program** to obtain the CloudXR runtime and client:
15+
16+
* **CloudXR Early Access**: `Join the NVIDIA CloudXR SDK Early Access Program <https://developer.nvidia.com/cloudxr-sdk-early-access-program/join>`_
17+
18+
Follow the steps in the confirmation email to get access to the CloudXR runtime container and client resources.
19+
20+
21+
Step 1: Start the CloudXR Runtime Container
22+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
23+
24+
#. Download the **CloudXR Runtime Container** from NVIDIA NGC. Version **6.0.1** is tested.
25+
26+
.. code-block:: bash
27+
28+
docker login nvcr.io
29+
docker pull nvcr.io/nvidia/cloudxr-runtime-early-access:6.0.1-webrtc
30+
31+
#. In a new terminal, start the CloudXR runtime container:
32+
33+
.. code-block:: bash
34+
35+
cd submodules/IsaacLab
36+
mkdir -p openxr
37+
38+
docker run -dit --rm --name cloudxr-runtime \
39+
--user $(id -u):$(id -g) \
40+
--gpus=all \
41+
-e "ACCEPT_EULA=Y" \
42+
--mount type=bind,src=$(pwd)/openxr,dst=/openxr \
43+
--network host \
44+
nvcr.io/nvidia/cloudxr-runtime-early-access:6.0.1-webrtc
45+
46+
47+
Step 2: Start Arena Teleop
48+
^^^^^^^^^^^^^^^^^^^^^^^^^^
49+
50+
In another terminal, start the Arena Docker container and launch the teleop session to verify the pipeline:
51+
52+
:docker_run_default:
53+
54+
.. code-block:: bash
55+
56+
python isaaclab_arena/scripts/imitation_learning/teleop.py \
57+
--enable_pinocchio \
58+
galileo_g1_locomanip_pick_and_place \
59+
--teleop_device openxr
60+
61+
Start the AR/XR session from the **AR** tab in the application window.
62+
63+
.. figure:: ../../../images/locomanip_arena_server.png
64+
:width: 100%
65+
:alt: Arena teleop with XR running (stereoscopic view and OpenXR settings)
66+
:align: center
67+
68+
Arena teleop session with XR running. Stereoscopic view (left) and OpenXR settings in the AR tab (right).
69+
70+
71+
Step 3: Build and Run the CloudXR.js WebServer
72+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
73+
74+
#. Download the `CloudXR.js with samples <https://catalog.ngc.nvidia.com/orgs/nvidia/resources/cloudxr-js-early-access?version=6.0.1-beta>`_, unzip and follow the included guide.
75+
76+
#. Start the CloudXR.js WebServer:
77+
78+
.. code-block:: bash
79+
80+
cd cloudxr-js-early-access_6.0.1-beta/release
81+
docker build -t cloudxr-isaac-sample --build-arg EXAMPLE_NAME=isaac .
82+
docker run -d --name cloudxr-isaac-sample -p 8080:80 -p 8443:443 cloudxr-isaac-sample
83+
84+
You can test from a local browser at ``http://localhost:8080/`` before connecting the Quest.
85+
86+
.. figure:: ../../../images/locomanip_cloudxr_js.png
87+
:width: 100%
88+
:alt: CloudXR.js Isaac Lab Teleop Client (connection and debug settings)
89+
:align: center
90+
91+
CloudXR.js Isaac Lab Teleop Client. Configure server IP and port, then press **Connect**. Adjust stream resolution and reference space in Debug Settings if needed.
92+
93+
Step 4: Setup and Connect from Meta Quest 3
94+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
95+
96+
#. On the host machine, update the firewall to allow traffic on these ports:
97+
98+
.. code-block:: bash
99+
100+
sudo ufw allow 49100/tcp
101+
sudo ufw allow 47998/udp
102+
103+
#. **Network**: Use a router with Wi-Fi 6 (5 GHz band). Connect the server via Ethernet and the Quest to the same router's Wi-Fi. See the `CloudXR Network Setup <https://docs.nvidia.com/cloudxr-sdk/latest/requirement/network_setup.html>`_ guide.
104+
105+
#. **Quest configuration**: On the Quest headset, configure insecure origins for HTTP mode (one-time setup):
106+
107+
* Open the Meta Quest 3 browser and go to ``chrome://flags``.
108+
* Search for ``insecure``, find ``unsafely-treat-insecure-origin-as-secure``, and set it to **Enabled**.
109+
* In the text field, enter your Arena host URL: ``http://<server-ip>:8080``.
110+
* Tap outside the text field; a **Relaunch** button appears. Tap **Relaunch** to apply.
111+
* After relaunch, return to ``chrome://flags`` and confirm the flag is still enabled and the URL is saved.
112+
113+
#. **Connect**: On the Quest, open the browser and go to ``http://<server-ip>:8080``. In Settings, enter the server IP, then press **Connect**. You should see the simulation and be able to teleoperate.
114+
115+
The browser will prompt for WebXR permissions the first time. Select **Allow**; the immersive session starts after permission is granted.
116+
117+
#. **Teleoperation Controls**:
118+
119+
* **Left joystick**: Move the body forward/backward/left/right.
120+
* **Right joystick**: Squat (down), rotate torso (left/right).
121+
* **Controllers**: Move end-effector (EE) targets for the arms.
122+
123+
.. hint::
124+
125+
Suggested sequence for the task:
126+
127+
#. Align your body with the robot.
128+
#. Walk forward (left joystick forward).
129+
#. Grab the box (controllers).
130+
#. Walk backward (left joystick back).
131+
#. Turn toward the bin (right joystick).
132+
#. Walk forward to the bin.
133+
#. Squat (right joystick down).
134+
#. Place the box in the bin (controllers).
135+
136+
137+
Step 5: Record with Quest 3
138+
^^^^^^^^^^^^^^^^^^^^^^^^^^^
139+
140+
#. **Recording**: When ready to collect data, run the recording script from the Arena container:
141+
142+
.. code-block:: bash
143+
144+
export DATASET_DIR=/datasets/isaaclab_arena/locomanipulation_tutorial
145+
mkdir -p $DATASET_DIR
146+
147+
# Record demonstrations with OpenXR teleop
148+
python isaaclab_arena/scripts/imitation_learning/record_demos.py \
149+
--xr \
150+
--device cpu \
151+
--enable_pinocchio \
152+
--dataset_file $DATASET_DIR/arena_g1_locomanipulation_dataset_recorded.hdf5 \
153+
--num_demos 10 \
154+
--num_success_steps 2 \
155+
galileo_g1_locomanip_pick_and_place \
156+
--teleop_device openxr
157+
158+
#. Complete the task (pick the brown box, place it in the blue bin) for each demo. Reset between demos. The script saves successful runs to the HDF5 file above.
159+
160+
161+
Optional: Replay Recorded Demos
162+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
163+
164+
To replay the recorded demos:
165+
166+
.. code-block:: bash
167+
168+
# Replay from the recorded HDF5 dataset
169+
python isaaclab_arena/scripts/imitation_learning/replay_demos.py \
170+
--device cpu \
171+
--dataset_file $DATASET_DIR/arena_g1_locomanipulation_dataset_recorded.hdf5 \
172+
--enable_pinocchio \
173+
galileo_g1_locomanip_pick_and_place

0 commit comments

Comments
 (0)