Support Quest teleop for G1 locomanipulation example (#350)

Support Quest3 for loco-manipulation teleop and record/replay

- What was the reason for the change?
Support Quest3 for loco-manipulation teleop for g1_pink example. This
allows proper data recording for the locomanipulation.
- What has been changed?
Adds openxr + g1_pink teleop support
Adds documention for locomanip data recording with Quest

Author: qianl-nv

docker/run_docker.sh

@@ -7,6 +7,9 @@ SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 
 WORKDIR="/workspaces/isaaclab_arena"
 
+# Default OpenXR directory shared with CloudXR runtime (lives in IsaacLab submodule)
+OPENXR_HOST_DIR="./submodules/IsaacLab/openxr"
+
 # Default mount directory on the host machine for the datasets
 DATASETS_HOST_MOUNT_DIRECTORY="$HOME/datasets"
 # Default mount directory on the host machine for the models

docs/images/locomanip_arena_server.png

@@ -1,7 +1,7 @@
 G1 Loco-Manipulation Box Pick and Place Task
 ============================================
 
-This example demonstrates the complete workflow for the **G1 loco-manipulation box pick and place task** in Isaac Lab - Arena, covering environment setup and validation, data generation, policy post-training, and closed-loop evaluation.
+This example demonstrates the complete workflow for the **G1 loco-manipulation box pick and place task** in Isaac Lab - Arena, covering environment setup and validation, teleoperation data collection (OpenXR with Meta Quest 3), data generation, policy post-training, and closed-loop evaluation.
 
 .. image:: ../../../images/g1_galileo_arena_box_pnp_locomanip.gif
    :align: center
@@ -56,7 +56,7 @@ including lower body locomotion, squatting, and bimanual manipulation.
 Workflow
 --------
 
-This tutorial covers the pipeline between creating an environment, generating training data,
+This tutorial covers the pipeline between creating an environment, collecting teleoperation demonstrations, generating training data,
 fine-tuning a policy (GR00T N1.6), and evaluating the policy in closed-loop.
 A user can follow the whole pipeline, or can start at any intermediate step
 by downloading the pre-generated output of the preceding step(s), which we provide
@@ -90,16 +90,18 @@ Workflow Steps
 Follow the following steps to complete the workflow:
 
 - :doc:`step_1_environment_setup`
-- :doc:`step_2_data_generation`
-- :doc:`step_3_policy_training`
-- :doc:`step_4_evaluation`
+- :doc:`step_2_teleoperation`
+- :doc:`step_3_data_generation`
+- :doc:`step_4_policy_training`
+- :doc:`step_5_evaluation`
 
 
 .. toctree::
    :maxdepth: 1
    :hidden:
 
    step_1_environment_setup
-   step_2_data_generation
-   step_3_policy_training
-   step_4_evaluation
+   step_2_teleoperation
+   step_3_data_generation
+   step_4_policy_training
+   step_5_evaluation

docs/images/locomanip_cloudxr_js.png

@@ -0,0 +1,176 @@
+Teleoperation Data Collection
+-----------------------------
+
+This workflow covers collecting demonstrations for the G1 loco-manipulation task using **Meta Quest 3** supported by **NVIDIA CloudXR**.
+
+This workflow requires several components to run:
+
+* **NVIDIA CloudXR Runtime**: Runs in a Docker container on your workstation and streams the Isaac Lab simulation to a compatible XR device. See the `CloudXR Runtime documentation <https://docs.nvidia.com/cloudxr-sdk/latest/usr_guide/cloudxr_runtime/index.html>`_.
+* **Arena Docker container**: Runs the Isaac Lab simulation and recording.
+* **CloudXR.js WebServer**: Meta Quest 3 and Pico 4 Ultra connect to Isaac Lab via the CloudXR.js WebXR client. See `CloudXR.js (Early Access) <https://docs.nvidia.com/cloudxr-sdk/latest/usr_guide/cloudxr_js/index.html>`_.
+
+.. note::
+
+   You must join the **NVIDIA CloudXR Early Access Program** to obtain the CloudXR runtime and client:
+
+   * **CloudXR Early Access**: `Join the NVIDIA CloudXR SDK Early Access Program <https://developer.nvidia.com/cloudxr-sdk-early-access-program/join>`_
+
+   Follow the steps in the confirmation email to get access to the CloudXR runtime container and client resources.
+
+
+Step 1: Start the CloudXR Runtime Container
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+#. Download the **CloudXR Runtime Container** from NVIDIA NGC. Version **6.0.1** is tested.
+
+   .. code-block:: bash
+
+      docker login nvcr.io
+      docker pull nvcr.io/nvidia/cloudxr-runtime-early-access:6.0.1-webrtc
+
+#. In a new terminal, start the CloudXR runtime container:
+
+   .. code-block:: bash
+
+      cd submodules/IsaacLab
+      mkdir -p openxr
+
+      docker run -dit --rm --name cloudxr-runtime \
+        --user $(id -u):$(id -g) \
+        --gpus=all \
+        -e "ACCEPT_EULA=Y" \
+        --mount type=bind,src=$(pwd)/openxr,dst=/openxr \
+        --network host \
+        nvcr.io/nvidia/cloudxr-runtime-early-access:6.0.1-webrtc
+
+
+Step 2: Start Arena Teleop
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In another terminal, start the Arena Docker container and launch the teleop session to verify the pipeline:
+
+:docker_run_default:
+
+.. code-block:: bash
+
+   python isaaclab_arena/scripts/imitation_learning/teleop.py \
+     --enable_pinocchio \
+     galileo_g1_locomanip_pick_and_place \
+     --teleop_device openxr
+
+Start the AR/XR session from the **AR** tab in the application window.
+
+.. figure:: ../../../images/locomanip_arena_server.png
+   :width: 100%
+   :alt: Arena teleop with XR running (stereoscopic view and OpenXR settings)
+   :align: center
+
+   Arena teleop session with XR running. Stereoscopic view (left) and OpenXR settings in the AR tab (right).
+
+
+Step 3: Build and Run the CloudXR.js WebServer
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+#. Download the `CloudXR.js with samples <https://catalog.ngc.nvidia.com/orgs/nvidia/resources/cloudxr-js-early-access?version=6.0.1-beta>`_, unzip and follow the included guide.
+
+#. Start the CloudXR.js WebServer:
+
+   .. code-block:: bash
+
+      cd cloudxr-js-early-access_6.0.1-beta/release
+      docker build -t cloudxr-isaac-sample --build-arg EXAMPLE_NAME=isaac .
+      docker run -d --name cloudxr-isaac-sample -p 8080:80 -p 8443:443 cloudxr-isaac-sample
+
+   You can test from a local browser at ``http://localhost:8080/`` before connecting the Quest.
+
+.. figure:: ../../../images/locomanip_cloudxr_js.png
+   :width: 100%
+   :alt: CloudXR.js Isaac Lab Teleop Client (connection and debug settings)
+   :align: center
+
+   CloudXR.js Isaac Lab Teleop Client. Configure server IP and port, then press **Connect**. Adjust stream resolution and reference space in Debug Settings if needed.
+
+Step 4: Setup and Connect from Meta Quest 3
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+#. On the host machine, update the firewall to allow traffic on these ports:
+
+   .. code-block:: bash
+
+      sudo ufw allow 49100/tcp
+      sudo ufw allow 47998/udp
+
+#. **Network**: Use a router with Wi-Fi 6 (5 GHz band). Connect the server via Ethernet and the Quest to the same router's Wi-Fi. See the `CloudXR Network Setup <https://docs.nvidia.com/cloudxr-sdk/latest/requirement/network_setup.html>`_ guide.
+
+#. **Quest configuration**: On the Quest headset, configure insecure origins for HTTP mode (one-time setup):
+
+   * Open the Meta Quest 3 browser and go to ``chrome://flags``.
+   * Search for ``insecure``, find ``unsafely-treat-insecure-origin-as-secure``, and set it to **Enabled**.
+   * In the text field, enter your Arena host URL: ``http://<server-ip>:8080``.
+   * Tap outside the text field; a **Relaunch** button appears. Tap **Relaunch** to apply.
+   * After relaunch, return to ``chrome://flags`` and confirm the flag is still enabled and the URL is saved.
+
+#. **Connect**: On the Quest, open the browser and go to ``http://<server-ip>:8080``. In Settings, enter the server IP, then press **Connect**. You should see the simulation and be able to teleoperate.
+
+   The browser will prompt for WebXR permissions the first time. Select **Allow**; the immersive session starts after permission is granted.
+
+#. **Teleoperation Controls**:
+
+* **Left joystick**: Move the body forward/backward/left/right.
+* **Right joystick**: Squat (down), rotate torso (left/right).
+* **Controllers**: Move end-effector (EE) targets for the arms.
+
+
+Step 5: Record with Quest 3
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+#. **Recording**: When ready to collect data, run the recording script from the Arena container:
+
+   .. code-block:: bash
+
+      export DATASET_DIR=/datasets/isaaclab_arena/locomanipulation_tutorial
+      mkdir -p $DATASET_DIR
+
+      # Record demonstrations with OpenXR teleop
+      python isaaclab_arena/scripts/imitation_learning/record_demos.py \
+        --device cpu \
+        --enable_pinocchio \
+        --dataset_file $DATASET_DIR/arena_g1_locomanipulation_dataset_recorded.hdf5 \
+        --num_demos 10 \
+        --num_success_steps 2 \
+        galileo_g1_locomanip_pick_and_place \
+        --teleop_device openxr
+
+#. Complete the task for each demo. Reset between demos. The script saves successful runs to the HDF5 file above.
+
+.. hint::
+
+   Suggested sequence for the task:
+
+   #. Align your body with the robot.
+   #. Walk forward (left joystick forward).
+   #. Grab the box (controllers).
+   #. Walk backward (left joystick back).
+   #. Turn toward the bin (right joystick).
+   #. Walk forward to the bin.
+   #. Squat (right joystick down).
+   #. Place the box in the bin (controllers).
+
+.. image:: ../../../images/g1_galileo_arena_box_pnp_locomanip.gif
+   :align: center
+   :height: 400px
+
+
+Step 6: Replay Recorded Demos (Optional)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To replay the recorded demos:
+
+.. code-block:: bash
+
+   # Replay from the recorded HDF5 dataset
+   python isaaclab_arena/scripts/imitation_learning/replay_demos.py \
+     --device cpu \
+     --dataset_file $DATASET_DIR/arena_g1_locomanipulation_dataset_recorded.hdf5 \
+     --enable_pinocchio \
+     galileo_g1_locomanip_pick_and_place

docs/pages/example_workflows/locomanipulation/index.rst

@@ -0,0 +1,124 @@
+Data Generation
+---------------
+
+This workflow covers annotating and generating the demonstration dataset using
+`Isaac Lab Mimic <https://isaac-sim.github.io/IsaacLab/main/source/overview/imitation-learning/teleop_imitation.html>`_.
+
+
+**Docker Container**: Base (see :doc:`../../quickstart/docker_containers` for more details)
+
+:docker_run_default:
+
+
+
+Step 1: Annotate Demonstrations
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This step describes how to annotate the demonstrations recorded in the preceding step
+so they can be used by Isaac Lab Mimic. For more details on Mimic annotation, see the
+`Isaac Lab Mimic documentation <https://isaac-sim.github.io/IsaacLab/main/source/overview/imitation-learning/teleop_imitation.html#annotate-the-demonstrationsl>`_.
+
+To skip this step, you can download the pre-annotated dataset from Hugging Face as described below.
+
+.. dropdown:: Download Pre-annotated Dataset (skip annotation step)
+   :animate: fade-in
+
+   These commands can be used to download the pre-annotated dataset,
+   such that the annotation step can be skipped.
+
+   To download run:
+
+   .. code-block:: bash
+
+      hf download \
+         nvidia/Arena-G1-Loco-Manipulation-Task \
+         arena_g1_loco_manipulation_dataset_annotated.hdf5 \
+         --repo-type dataset \
+         --local-dir $DATASET_DIR
+
+To start the annotation process, run the following command:
+
+.. code-block:: bash
+
+   python isaaclab_arena/scripts/imitation_learning/annotate_demos.py \
+     --device cpu \
+     --input_file $DATASET_DIR/arena_g1_locomanipulation_dataset_recorded.hdf5 \
+     --output_file $DATASET_DIR/arena_g1_locomanipulation_dataset_annotated.hdf5 \
+     --enable_pinocchio \
+     --mimic \
+     galileo_g1_locomanip_pick_and_place
+
+Follow the instructions described on the CLI to complete the annotation.
+
+
+
+Step 2: Generate Augmented Dataset
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Isaac Lab Mimic generates additional demonstrations from the annotated demonstrations
+by applying object and trajectory transformations to introduce data variations.
+
+This step can be skipped by downloading the pre-generated dataset from Hugging Face as described below.
+
+.. dropdown:: Download Pre-generated Dataset (skip data generation step)
+   :animate: fade-in
+
+   These commands can be used to download the pre-generated dataset,
+   such that the data generation step can be skipped.
+
+   .. code-block:: bash
+
+      hf download \
+         nvidia/Arena-G1-Loco-Manipulation-Task \
+         arena_g1_loco_manipulation_dataset_generated.hdf5 \
+         --repo-type dataset \
+         --local-dir $DATASET_DIR
+
+Generate the dataset:
+
+.. code-block:: bash
+
+   # Generate 100 demonstrations
+   python isaaclab_arena/scripts/imitation_learning/generate_dataset.py \
+     --headless \
+     --enable_cameras \
+     --mimic \
+     --input_file $DATASET_DIR/arena_g1_loco_manipulation_dataset_annotated.hdf5 \
+     --output_file $DATASET_DIR/arena_g1_loco_manipulation_dataset_generated.hdf5 \
+     --generation_num_trials 100 \
+     --device cpu \
+     galileo_g1_locomanip_pick_and_place \
+     --object brown_box \
+     --embodiment g1_wbc_pink
+
+Data generation takes 1-4 hours depending on your CPU/GPU.
+You can remove ``--headless`` to visualize during data generation.
+
+
+Step 3: Validate Generated Dataset (Optional)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To visualize the data produced, you can replay the dataset using the following command:
+
+.. code-block:: bash
+
+   python isaaclab_arena/scripts/imitation_learning/replay_demos.py \
+     --device cpu \
+     --enable_cameras \
+     --dataset_file $DATASET_DIR/arena_g1_loco_manipulation_dataset_generated.hdf5 \
+     galileo_g1_locomanip_pick_and_place \
+     --object brown_box \
+     --embodiment g1_wbc_pink
+
+You should see the robot successfully perform the task.
+
+.. figure:: ../../../images/g1_locomanip_pick_and_place_task_view.png
+   :width: 100%
+   :alt: G1 Locomanip Pick and Place Task View
+   :align: center
+
+   IsaacLab Arena G1 Locomanip Pick and Place Task View
+
+.. note::
+
+   The dataset was generated using CPU device physics, therefore the replay uses ``--device cpu`` to ensure reproducibility.

docs/pages/example_workflows/locomanipulation/step_2_teleoperation.rst

@@ -16,7 +16,7 @@ Once inside the container, set the dataset and models directories.
     export MODELS_DIR=/models/isaaclab_arena/locomanipulation_tutorial
 
 Note that this tutorial assumes that you've completed the
-:doc:`preceding step (Data Generation) <step_2_data_generation>` or downloaded the pre-generated dataset.
+:doc:`preceding step (Data Generation) <step_3_data_generation>` or downloaded the pre-generated dataset.
 
 .. dropdown:: Download Pre-generated Dataset (skip preceding steps)
    :animate: fade-in

docs/pages/example_workflows/locomanipulation/step_3_data_generation.rst

@@ -17,7 +17,7 @@ Once inside the container, set the dataset and models directories.
     export MODELS_DIR=/models/isaaclab_arena/locomanipulation_tutorial
 
 Note that this tutorial assumes that you've completed the
-:doc:`preceding step (Policy Training) <step_3_policy_training>` or downloaded the
+:doc:`preceding step (Policy Training) <step_4_policy_training>` or downloaded the
 pre-trained model checkpoint below:
 
 .. dropdown:: Download Pre-trained Model (skip preceding steps)

…omanipulation/step_3_policy_training.rst …omanipulation/step_4_policy_training.rstdocs/pages/example_workflows/locomanipulation/step_3_policy_training.rst renamed to docs/pages/example_workflows/locomanipulation/step_4_policy_training.rst docs/pages/example_workflows/locomanipulation/step_3_policy_training.rst renamed to docs/pages/example_workflows/locomanipulation/step_4_policy_training.rst

@@ -133,7 +133,13 @@ def get_teleop_device_cfg(self, device: type["TeleopDeviceBase"], embodiment: ob
         retargeter_key_str = retargeter_registry.convert_tuple_to_str(retargeter_key)
         retargeter = retargeter_registry.get_component_by_name(retargeter_key_str)()
         retargeter_cfg = retargeter.get_retargeter_cfg(embodiment, sim_device=device.sim_device)
-        retargeters = [retargeter_cfg] if retargeter_cfg is not None else []
+        # Handle both single retargeter and list of retargeters
+        if isinstance(retargeter_cfg, list):
+            retargeters = retargeter_cfg
+        elif retargeter_cfg is not None:
+            retargeters = [retargeter_cfg]
+        else:
+            retargeters = []
         device_cfg = device.get_device_cfg(retargeters=retargeters, embodiment=embodiment)
         return DevicesCfg(
             devices={