Unity ROS Teleoperation Project

This repo contains a series of components for Unity XR teleoperation with ROS integration. It is designed to be run on a Quest 3 VR headset, and support OpenXR, hand tracking, Unity input system, and is built with Unity 2022.3.12f1.

For information on setting up Unity and opening this project Unity Quickstart, and for Quest information see Quest Quickstart.

Quickstart

On the ROS side, the custom TCP Endpoint needs to be run somewhere on the ROS network. This node is pretty lightweight so it can be run directly on a robot. Once the node is running the device running this app needs to connect to the ROS network, and the IP of the device running the TCP Endpoint needs to be set in the menu (see Menu for more information). If everything works, the menu should turn green, and data should be streaming between the app and ROS.

Components

Component	Description	Location	Preview
Audio Streamer	Implements bidirectional audio stream over ROS	Assets/Components/AudioStreamer
Camera Viewer	Renders a ROS image stream to a floating image window	Assets/Components/CameraView
Grid Map	Renders a 2.5D grid map such as elevation maps from Anymal	Assets/Components/GridMap
Hands	Hand tracking and pose publishing over ROS, compatible with Ability hand models	Assets/Components/Hands
Haptic	Bhaptic glove support plus controller haptics	Assets/Components/Haptics
Headset Publisher	Publishes headset and hand poses on TF and Pose	Assets/Components/HeadsetPublisher
Lidar	GPU rendering for LiDAR and PointCloud2 point viz from ROS	Assets/Components/Lidar
Markers	Displays various ROS markers in the scene (supports points, cubes, spheres)	Assets/Components/Markers
Menu	Palm menu for interaction and toggling	Assets/Components/Menu
NeRFViewer	Handheld viewer for rendering NeRFs and scene interaction	Assets/Components/NeRFViewer
PathStreamer	Displays nav paths with a line and series of arrows	Assets/Components/PathStreaming
PosePublisher	Publishes poses and Joystick commands for locomotion	Assets/Components/PosePublisher
PoseStreamer	Displays stamped poses	Assets/Components/PoseStreaming
Robots	Robot model manager, holds the meshes, materials and the scripts to swap between Anymal, ALMA, Tytan, Standalone Dynaarm and Franka Panda	Assets/Components/Robots
Service Caller	Creates a floating button that triggers an Empty service call when pressed	Assets/Components/ServiceCaller
Stereo	Stereo camera rendering, renders to each eye for human depth perception	Assets/Components/StereoImage
TF	Updates for the Unity-ROS TF system for managing robots and reorientation including publishing headset and hands over TF	Assets/Components/TFSystem
TF Viz	Debug viewer to show currently streamed TF links	Assets/Components/TFViz
Voxblox	Voxel mesh rendering	Assets/Components/VoxBlox
VR Debug	Debugging tools for VR, namely a console	Assets/Components/VRDebug
VR Streamer	Streams a the VR view to a ROS topic (w/o AR view)	Assets/Components/VRStreamer

Scenes

In general the scenes should have a few objects by default:

• Light source (usually the default directional light)
• MR Interaction Setup (this enables AR/VR support and acts as a camera)
• Debug canvas (will autolink to the menu and shows debug messages)
• Palm Menu (menu to interact and toggle things with)
• Root (The roof of the TF/object tree and tagged at 'root')

---

Minor Version History

0.1.0

May need to reset the repo as LFS has been removed from history

• Added 2D version of the app
• Refactored new Manager-Streamer system for standardized message visualizations
• Added new visualizations
  • Markers (without meshes and text)
  • Paths
  • Stamped Poses
  • TF Visualization for frames
  • Grid Maps/Elevation Maps
• Added floating button to trigger Empty service calls
• Redid TF system so the root frame can be dynamic and not locked to odom (also allows functioning without a robot model)
• Improved Pose publishing and allows to work with 2D interface
• Added debug menus to display force
• Overhauled PointCloud rendering for better performance and shader keywords for customized coloring
• Added mesh shaders for efficently render large numbers of procedural meshes
• Improved layout serialization allowing for saving of custom configurations

0.0.9

• Simplified dynaarm model (down to 98 draw calls and 330k tris)
• Fixed TF pose publishing so it should work even without a model
• Removed some duplicated publishers/gameobjects

0.0.8

• Added versioning with display in debug mode
• Added tytan, standalone dynaarm, and franka panda robots
• Updated dynaarm for newer model (may still need simplification)
• Added robot manager for switching between robots

---

Citing

If you use this project in your work please cite this paper:

@INPROCEEDINGS{wildersmith2024rfteleoperation,
        author={Wilder-Smith, Maximum and Patil, Vaishakh and Hutter, Marco},
        booktitle={2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, 
        title={Radiance Fields for Robotic Teleoperation}, 
        year={2024},
        pages={13861-13868},
        doi={10.1109/IROS58592.2024.10801345}
}