CI Wheels 22478032608.1

Automated wheel artifacts for bd0422c

CI Wheels 22477731015.1

Automated wheel artifacts for ccc0f92

Release 0.3.15

Isaac Teleop (TeleopCore) Release Notes

Release 0.3.15 (vs 0.2)

Release branch: release/0.3.x

---

Highlights

• Project rename: TeleopCore is now Isaac Teleop — the unified framework for high-fidelity ego-centric and robotics data collection. Documentation and branding have been updated accordingly.
• Retargeting framework: Port of Isaac Lab retargeters with a type system, source nodes, and examples (gripper, SE3, dex bimanual, G1 tri-hand, locomotion, reordering). Retargeter tuning UI with IMGUI for live tuning.
• Camera & recording: Generic Camera Device Plugin interface; Oak-D plugin; MCAP recorder integrated into deviceio session; generic per-frame metadata in camera schema.
• Full body & locomotion: Pico full body tracking schema and testbed; locomotion schema and bindings compatible with Isaac OS Lower Body Controller.
• CloudXR: Runtime updates to 6.1.0 (pid3/pid4/rc2); tensor extension support; streamlined Docker setup and env handling; Quest3 as default device profile.
• Python & packaging: Python 3.10, 3.11, and 3.12 support; PEP 440 versioning; multi-version wheel builds in CI; git commit count in versioning.

---

New Features

• Retargeting

  • Retargeter type system and interface; port of Isaac Lab retargeters (SE3, gripper, dex, combined).
  • Source nodes and enums for source node identifiers.
  • Examples: controller + SE3, gripper, dex bimanual, G1 tri-hand, locomotion, full bimanual reordering.
  • Retargeter tuning UI (IMGUI) for live tuning.
  • Optional retargeting dependencies in build.

• Camera & device plugins

  • Generic Camera Device Plugin interface; Oak-D camera plugin (DepthAI) for recording.
  • MCAP recorder integrated into deviceio_session with configurable recording path; Python example for session with MCAP recording.
  • Generic per-frame metadata in camera.fbs (#88).
  • hands.fbs for tracking both hands to align with hand tracker logic.

• Full body & locomotion

  • Data schema and bindings for Pico full body tracking.
  • Body tracking testbed for Pico WebXR body tracking extension (cxrjs/bodytracking example).
  • Locomotion schema and bindings compatible with Isaac OS Lower Body Controller operator.

• Session & OpenXR

  • Create OpenXRSessionHandles from Python (#81).
  • Teleop session manager package: helpers, config, and teleop_session_example.
  • Shared core::OpenXRSession reuse across plugins; OpenXR math helpers moved to oxr_utils.
  • CXR tensor extension headers (Generic Tensor Collection / push tensor) for plugins.

• Device I/O & schemas

  • Device output for Generic3AxisPedalOutput (e.g. Logitech-style pedals).
  • Timestamps on device data; controller tracker uses FlatBuffer schema.
  • Tensor and TensorGroup classes with type validation and tests; tensor types for retargeting engine I/O.

• Build & developer experience

  • Devcontainer configuration for Isaac Teleop (#66).
  • Teleop utils for building teleop apps.
  • TELEOPCORE_DISABLE_CXR_ENV_CHECKS to disable CloudXR env checks when needed.

---

Improvements

• DeviceIO & trackers

  • Tracker creation refactor: initialize → create_tracker; implementations retrieved from session; removed is_initialized() from tracker interfaces (#33, #34).
  • Device plugin reuses controller from deviceio; rate-limited warnings in DeviceIOSession to reduce log spam.
  • deviceio C++ files renamed to snake_case for consistency.

• CloudXR

  • CloudXR env setup: fail early on missing env; use NV_CXR_RUNTIME_DIR (replacing XDG_RUNTIME_DIR where appropriate); ensure OpenXR env vars (XR_RUNTIME_JSON, etc.) are set in oxr_session.
  • CloudXR Docker: run_cloudxr.sh can download Web SDK from NGC, build containers, and start services; cloudxr_image_ops.sh for flexible image load/save/pull with container_tag.
  • Migrated CloudXR JS SDK download from manual PID to NGC CLI; one-time NGC setup documented.
  • README improvements for CXR Runtime setup; HAProxy Dockerfile tidied (permissions).

• Build & CI

  • Hunter cache (.hunter) to improve build speed for OAKD dependencies (#85).
  • ccache (Ubuntu) and sccache (Windows) support (#29).
  • CMake cleanup: redundant checks removed, inc aligned with target names, deviceio_py_utils library (#40).
  • DepthAI/CMake: policy version set for compatibility with CMake ≥ 4.0 (#60).
  • Ubuntu CI uses Ubuntu-22.04; GPU runner for GitHub Actions; native tests run in CI.
  • Publish-wheel job for tagged releases (#70); multi-Python wheel builds (#89).
  • Docker-based test fixture for CloudXR tests (local and CI); run_tests_with_cloudxr.sh improvements and documentation.

• Versioning & deps

  • Single consolidated VERSION file; version includes git commit count (#83); bump to 0.3.x.
  • PEP 440 versioning for alpha, rc, dev, local, and final releases (#87).
  • NumPy 1.22+ (aligned across docs); optional update to 2.4.0 in examples.
  • get_home_dir improved (Windows: USERPROFILE; runtime error instead of /tmp fallback).

• Security & supply chain

  • npm install runs with --ignore-scripts to reduce supply chain risk (#9da1d58).
  • HAProxy config ownership set to root to address security hotspot.

• Oak-D / camera

  • Default size switched to match camera resolution to avoid unnecessary scaling and preserve FOV (#74).

• LeRobot

  • Recorder fixes; dataset written relative to recorder.py location.

• Misc

  • OpenXR SDK compatibility via sanitizers-cmake (#43).
  • Removed internal references (#72); README and doc cleanups.
  • Issue templates updated; unnecessary README sections removed.
  • UV: specify Python version in Dockerfile.test; remove uv.lock files from repo.

---

Bug Fixes

• Use get_required_extensions correctly for OpenXR.
• Fix typo in load_cloudxr_images.sh.
• Remove duplicate file and ctor definitions from OpenXRSessionHandles; add default initializers (#67).
• LeRobot recorder fixes and path handling.

---

API / Behavior Changes

• Trackers: initialize() replaced by create_tracker(); is_initialized() removed from tracker interfaces and implementations.
• Versioning: VERSION file consolidated; format follows PEP 440; git commit count used in version string.
• CloudXR: Prefer NV_CXR_RUNTIME_DIR; stricter env checks unless disabled via TELEOPCORE_DISABLE_CXR_ENV_CHECKS.
• Plugin utils: Controllers and session helpers removed from plugin_utils (logic moved/reused in deviceio/plugins where applicable).

---

Dependencies & Requirements

• Python: 3.10, 3.11, and 3.12 supported.
• NumPy: 1.22 or newer (documented; 2.4.0 used in some examples).
• CloudXR: Runtime 6.1.0 series (pid3, pid4, rc2, or webrtc patch as referenced in commits).
• Build: Ubuntu 22.04 for build-ubuntu workflow; CMake policy compatibility for DepthAI with CMake ≥ 4.0.

---

Known Limitations / Notes

• Oak-D plugin: Windows build skipped in CI due to ffmpeg-related constraints (see #60).
• CloudXR tests require GPU and optional Docker/CloudXR setup; see run_tests_with_cloudxr.sh and README.

---

Generated from diff: release/0.2..release/0.3.x (TeleopCore repo).

Release 0.2.0

Overview

This release introduces new features including a device plugin system, data schemas
with FlatBuffer serialization, and major improvements to the hand/head tracking systems. The
codebase has been modernized with C++20 support, improved Python packaging, and better code
quality through clang-format enforcement.

---

🚀 New Features

Plugin Manager System

• Plugin Manager API - New plugin architecture for discovering and managing external plugin
  binaries with YAML-based metadata configuration
• Plugin health monitoring - Added check_health API that throws PluginCrashException if
  the plugin process crashes
• Manus Glove Plugin - Full integration with Manus SDK for glove-based hand tracking with
  controller pose fusion
• Controller Synthetic Hands Plugin - Generate synthetic hand poses from VR controllers

Schema & Data Serialization

• FlatBuffer-based schemas for efficient data serialization:
  • pose.fbs - Pose representation with position and quaternion
  • tensor.fbs - Tensor message for multi-dimensional data
  • hand.fbs - Hand pose schema with joint data
  • head.fbs - Head tracking schema
• Python bindings for all schemas with comprehensive unit tests
• C++ Object API for working with schema data

Device I/O Module

• Renamed and reorganized xrio module to deviceio for clarity
• Controller Tracker - New tracking support for VR controllers
• Head Tracker - Dedicated head pose tracking component
• Integrated FlatBuffer schemas for head and hand pose data

Examples & Documentation

• LeRobot Example - New example for robot learning data recording and visualization
• Camera Streaming Example - Standalone GStreamer-based camera streaming with OAK-D support
  (host/robot Docker configs)
• Improved README with better setup and build instructions

---

🔧 Improvements

Build System & Dependencies

• Switched from Git submodules to CMake FetchContent for dependency management
• Upgraded to C++20 to enable designated initializers
• clang-format enforcement in CI for consistent code style
• Improved Python packaging:
  • Locked Python version in pyproject.toml
  • Explicit Python version and ABI in wheel filenames
  • Auto-generated .pyi stub files for Python bindings
• Use mainline yaml-cpp instead of fork

Code Quality

• RAII improvements - Replaced OpenXRHandle with std::unique_ptr
• RAII for XR core functions - Better resource management
• Namespace reorganization - Renamed from oxr to core
• Extensive refactoring of hand tracker and session APIs

CI/CD

• Added clang-format checks to Linux builds
• CI optimizations to avoid exceeding usage limits
• Fixed Python test paths in Ubuntu build workflow

CloudXR Integration

• Enabled push device support in CloudXR container
• Updated web app and WSS proxy Dockerfiles
• Run nginx with non-root user for security
• Fixed HAProxy configuration permissions

---

⚠️ Breaking Changes

• xrio module renamed to deviceio - update import paths accordingly
• TeleopSession API replaced with DeviceIOSession
• Namespace changed from oxr:: to core:: in C++ code

---

🙏 Contributors

Thanks to all contributors who made this release possible!

Release 0.1.0

Initial commit for Isaac Teleop Core 0.1.0