GaussianDream is a feed-forward 3D Gaussian world model for robotic manipulation. The core implementation lives under the gaussiandream Python package, while legacy checkpoint/config identifiers and external asset paths are kept compatible with upstream OpenPI releases.
git clone https://github.com/TuojingAI/GaussianDream.git
cd GaussianDream
uv syncThe package metadata and implementation import namespace are both gaussiandream.
Third-party simulators and hardware stacks are not vendored in this repo. Please clone them yourself under third_party/ before running the corresponding track:
mkdir -p third_party
git clone https://github.com/Lifelong-Robot-Learning/LIBERO.git third_party/libero
git clone https://github.com/robocasa/robocasa.git third_party/robocasa
git clone https://github.com/ARISE-Initiative/robosuite.git third_party/robosuite
git clone https://github.com/Physical-Intelligence/aloha.git third_party/alohaOptional cache paths:
export GAUSSIANDREAM_DATA_HOME=<CACHE_DIR>
export CHECKPOINT_DIR=<CHECKPOINT_DIR>
export DATA_ROOT=<DATA_ROOT>OPENPI_DATA_HOME is still supported as a fallback for users with existing caches.
GaussianDream includes three evaluation paths:
- real-robot / runtime clients built around the shared policy server and
gaussiandream-client - LIBERO simulation evaluation in
examples/libero/ - RoboCasa simulation evaluation in
examples/robocasa/
Start a policy server from a checkpoint:
uv run scripts/serve_policy.py \
--port 8010 \
policy:checkpoint \
--policy.config gaussiandream_libero \
--policy.dir <CHECKPOINT_DIR>/<config>/<exp>/<step>For RoboCasa checkpoints, use --policy.config gaussiandream_robocasa and a matching checkpoint directory.
See examples/libero/README.md for setup, training, and evaluation commands.
Typical training flow:
export LIBERO_DATA_WITH_DEPTH_ROOT=<LIBERO_DATA_WITH_DEPTH_ROOT>
export LIBERO_FLOW_ROOT=<LIBERO_FLOW_ROOT>
export GAUSSIANDREAM_PRETRAINED_DIR=<PRETRAINED_MODEL_DIR>
uv run scripts/compute_norm_stats.py --config-name gaussiandream_libero
CUDA_VISIBLE_DEVICES=0 uv run scripts/train_pytorch.py \
gaussiandream_libero \
--exp-name gaussiandream_libero_runTypical flow:
uv run scripts/serve_policy.py --env LIBERO
python examples/libero/main.py --args.task-suite-name libero_10See examples/robocasa/README.md for conda setup, training, RoboCasa assets, and rollout commands.
Typical training flow:
export ROBOCASA_H50_ROOT=<ROBOCASA_H50_ROOT>
export GAUSSIANDREAM_PRETRAINED_DIR=<PRETRAINED_MODEL_DIR>
uv run scripts/compute_norm_stats.py --config-name gaussiandream_robocasa
CUDA_VISIBLE_DEVICES=0 uv run scripts/train_pytorch.py \
gaussiandream_robocasa \
--exp-name gaussiandream_robocasa_runTypical flow:
uv run scripts/serve_policy.py \
--port 8010 \
policy:checkpoint \
--policy.config gaussiandream_robocasa \
--policy.dir <CHECKPOINT_DIR>/<config>/<exp>/<step>
python examples/robocasa/main.py \
--host 127.0.0.1 \
--port 8010 \
--env-name PnPCounterToCab \
--prompt "pick and place from counter to cabinet"For RoboCasa H50 temporal evaluation, use examples/robocasa/eval_h50_temporal.py after installing the RoboCasa client environment.
See examples/aloha_real/README.md for the ALOHA hardware runtime, dataset conversion flow, and training examples.
Typical data pipeline:
python scripts/run_real_robot_pipeline.py \
--raw-dir <RAW_PICKLE_DIR> \
--lerobot-root <LEROBOT_ROOT> \
--repo-id local/gaussiandream_aloha \
--config-name gaussiandream_aloha \
--exp-name gaussiandream_aloha_runTypical serving flow for a trained ALOHA checkpoint:
uv run scripts/serve_policy.py \
--env ALOHA \
--port 8000 \
policy:checkpoint \
--policy.config gaussiandream_aloha \
--policy.dir <ALOHA_CKPT_DIR>/gaussiandream_aloha/<exp>/<step>GaussianDream currently exposes three main training configs:
gaussiandream_libero: LIBERO simulation training with Gaussian/world-model supervision.gaussiandream_robocasa: RoboCasa H50 training initialized from the LIBERO PyTorch checkpoint.gaussiandream_aloha: real-robot / ALOHA fine-tuning on your converted LeRobot dataset.
The common pattern is:
uv run scripts/compute_norm_stats.py --config-name <config_name>
CUDA_VISIBLE_DEVICES=0 uv run scripts/train_pytorch.py <config_name> --exp-name <run_name>Current environment variables expected by the released configs:
export GAUSSIANDREAM_PRETRAINED_DIR=<PRETRAINED_MODEL_DIR>
export LIBERO_DATA_WITH_DEPTH_ROOT=<LIBERO_DATA_WITH_DEPTH_ROOT>
export LIBERO_FLOW_ROOT=<LIBERO_FLOW_ROOT>
export ROBOCASA_H50_ROOT=<ROBOCASA_H50_ROOT>
export GAUSSIANDREAM_ALOHA_CKPT_DIR=<ALOHA_CKPT_ROOT>
export GAUSSIANDREAM_ALOHA_CKPT_DIR_TORCH=<ALOHA_CKPT_ROOT_TORCH>
export HF_LEROBOT_HOME=<LEROBOT_ROOT>The released gaussiandream_robocasa config currently expects the initialization file
$GAUSSIANDREAM_PRETRAINED_DIR/pi05_libero.safetensors, so the recommended order is:
- train or place the LIBERO PyTorch checkpoint first;
- fine-tune RoboCasa from that checkpoint;
- serve the resulting checkpoint with
scripts/serve_policy.pyfor evaluation.
Checkpoints, datasets, and complete reproduction instructions are coming soon. Large artifacts such as datasets, checkpoints, rendered videos, logs, and experiment outputs are intentionally not tracked in git.
If you have any question, please email zijianzhang821@gmail.com.
Appreciate the following works for their great contributions:
- VGGT: Inspires our 3D-aware visual representation design.
- OpenPI and π0: Serve as the foundation for our policy backbone and codebase.
- 3D Gaussian Splatting: Inspires our 3D Gaussian world modeling design.
- LIBERO and RoboCasa: Serve as the benchmarks for simulation training and evaluation.
If you find this work useful in your research, please cite:
@article{zhang2026gaussiandream,
title={GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation},
author={Zhang, Zijian and Jiang, Yuqing and Cheng, Qian and Li, Xiaofan and Liu, Si and Zhao, Ding and Luo, Ping and Zhou, Weitao and Yu, Haibao},
journal={arXiv preprint arXiv:2605.20752},
year={2026}
}