This repository demonstrates how Rerun can be used to process and use the BitRobot/HIW-500 dataset (Unitree G1 bimanual teleoperation, "Humanoid In the Wild"). We show how Rerun helps preprocess and augment the raw ROS2 MCAP episodes into per-episode recordings, view them, query across episodes, and train on them.
Work in progress: this repository is evolving — some capabilities (e.g. training) are still being built out, and we'll add more over time.
Note: this demo uses Pixi — you can get it here.
We use a subset of the dataset for this demo — three "Move the pillow…" takes, 29 short episodes
(~5.6 GB). Download them into HIW-500/:
pixi run download_sample_episodesThe downloader deliberately resolves only these subtrees before downloading their files. Avoid using
hf download --include for this dataset: the Hugging Face CLI enumerates the entire repository
(over 90,000 entries) before applying the pattern, which can look like a hang.
This repository was built using Rerun's agent skills. You can get them via
npx skills add rerun-io/rerun.
The dataset consists of episodes, each a ROS2 MCAP recording of a Unitree G1 with two
Dex1 single-DoF grippers performing a manipulation task. The MCAPs carry custom ROS2
message types alongside the standard ones, and a sibling info.json sidecar holds the task
label, subtask boundaries (ns timestamps), and scene id.
| Topic | Rate | Meaning |
|---|---|---|
/camera/head/image/compressed |
30 Hz | stereo head cam, 1280×480 (two 640×480 side-by-side) |
/camera/{left,right}_wrist/image/compressed |
30 Hz | wrist cams, 640×480 |
/wbc_lerobot |
50 Hz | LeRobot payload (JSON): bimanual EE-space ee_state (obs), ee_action, gripper_controls, pivot |
/stamped/lowstate |
100 Hz | 35-slot Unitree motor array (first 29 real): q/dq/ddq/tau_est/temp + IMU |
/stamped/lowcmd |
100 Hz | motor commands (q/dq/tau/kp/kd) |
/stamped/secondary_imu |
100 Hz | quaternion/gyro/accel/rpy |
/stamped/dex1/{left,right}/{cmd,state} |
100 Hz | per-gripper joint |
/lf/odommodestate |
20 Hz | base odometry: position, velocity, rpy, foot forces |
/annotation |
static | task label string |
/wbc_lerobot.ee_state / ee_action are 12 floats = [left 6][right 6], each a 6-DoF
position(xyz) + rotation-vector(3).
Each episode is processed into several Rerun recordings (.rrd) that share a recording_id, so
the viewer/catalog stacks them as layers of one logical recording: a faithful base layer
plus derived layers (robot model, odometry, cameras, metadata properties). Layers let us
flexibly augment the dataset — add, replace, or re-run a derived view without touching the
others, and overlay them only when wanted.
Build every layer for every episode in one command:
pixi run process # all episodes under HIW-500/
pixi run process <ep.mcap> # a single episodeEach layer also has its own task (process_base, process_urdf, process_odom,
process_cameras, process_properties) writing its own .rrd, so any layer can be run or
re-run independently —
fix the camera layer and rebuild just that, or add a layer to already-processed episodes. Each
task takes the same optional single-episode argument. (After an entity-layout change, re-run
register so the catalog picks it up.)
hiw_500/preprocess.py — a faithful conversion of the raw streams, no kinematics. A single
McapReader stream shaped by lenses: cameras decode to EncodedImage (the stereo head is split
into /camera/head/{left,right}), the custom homies/* / unitree_go/* messages become
per-joint and end-effector scalars/transforms (/state/…, /cmd/…, /lerobot/…), and
info.json becomes the /episode + /task/subtask sidecar. Entities keep the MCAP-native
timelines.
hiw_500/urdf_layer.py — the animated Unitree G1 mesh, driven by forward kinematics from the
/stamped/lowstate joint positions (emitted to /robot/transforms). Its 29 revolute joints
match the documented Unitree motor order. (The Dex1 finger joints stay at rest — no 1:1 mapping
in the URDF.)
The URDF is not part of the HF dataset. We use g1_29dof_mode_15_with_dex1_1.urdf from
Unitree's unitree_ros
(the Dex1 variant), vendored with its meshes under urdf/g1/.
hiw_500/odom_layer.py — connects the robot to the world. The URDF layer roots the G1 at
pelvis, so on its own it animates in place; this layer adds the time-varying odom → pelvis
transform from /lf/odommodestate so the whole robot moves through the scene.
hiw_500/camera_layer.py — places the head camera in 3D. The head D435 is a fixed joint in the
URDF, so its mount is already known; this layer adds the optical-frame transform plus the
Pinhole intrinsics (from a vendored D435 calibration at calibration/) so the head image
projects onto its image plane (both stereo eyes). The wrist cameras stay 2D — the dataset
provides no camera→robot (hand-eye) calibration for them.
TODO / caveat: the calibration in
calibration/was grabbed from a different episode'scalibration/params/on HuggingFace (our sample episodes don't ship one). It may not be the right calibration for these cameras — cleaning this up (using each episode's own calibration) is a known TODO.
hiw_500/properties_layer.py — per-episode metadata logged as recording properties, which the
catalog surfaces as columns to filter, sort, and search on: task, task_group,
duration_sec, num_subtasks, subtask_labels, robot. Values come from info.json, the
dataset path, and a constant.
Each episode is one logical recording split across five .rrd layers (base, urdf,
odom, cameras, properties) that share a recording_id.
Open the recordings straight in the viewer — same recording_ids merge into one recording per
episode, so all layers line up:
rerun rrds/*.rrd # every episode
rerun rrds/<id>*.rrd # one episode's layersGood for a quick look. No server, no registration.
hiw_500/blueprint.py builds a default viewer layout (a 3D scene in the odom frame, the four
camera streams, a task log, and joint / end-effector plots) and saves it to blueprints/hiw_500.rbl:
pixi run blueprintregister (below) installs it as the dataset's default blueprint, so every episode opens with
this layout instead of an auto-generated blob.
The more interesting use of a dataset like this is a catalog: episodes as queryable
segments with named layers, the way the cloud ingestion flow works. Layer names are assigned at
registration time — so don't point rerun server -d rrds at the folder, it loads every file
under the default layer base and fails with Layer 'base' already exists. Register instead:
pixi run serve # start the in-memory Rerun catalog on :51234 (leave running)
pixi run register # in another shell: register all episodes as the `hiw_500` datasetregister (hiw_500/catalog.py) creates the dataset, attaches each episode's RRDs as its
base/urdf/odom/cameras/properties layers (paired by recording_id; the properties
layer becomes per-segment columns), and installs blueprints/hiw_500.rbl as the default
blueprint. Then connect the viewer to rerun+http://127.0.0.1:51234, where you can browse, sort,
filter, and query across episodes rather than open them one at a time.
hiw-500-1080.mp4
- Rerun Learn — a hands-on course covering a similar example.
- Chunk processing API — the reader + lens pipeline this conversion is built on.
- Lenses — reshaping/deriving components in-stream.
robot_data_preprocessingexample — end-to-end MCAP + URDF + sidecar pipeline.