Feat/Quest VR teleoperation stack

[ruthwikdasyam]

Summary

• Full Meta Quest 3 teleoperation pipeline: WebXR capture (~80Hz) → Deno WebSocket/LCM bridge → Python control module (50Hz)
• QuestTeleopModule base with upgradable/overridable hooks (_handle_engage, _should_publish, _get_output_pose, _publish_msg, _publish_button_state) for subclassing different teleop behaviors
• QuestButtons(UInt32) bitmask with readable attribute accessors (buttons.left_x, buttons.right_grip)
• QuestControllerState preserves full analog fidelity from Joy messages (trigger/grip as floats, thumbstick axes)
• WebXR → robot coordinate frame transform (webxr_to_robot)
• Subclasses: ArmTeleopModule (toggle engage), TwistTeleopModule (twist output), VisualizingTeleopModule (Rerun debug)
• Pre-composed blueprints over LCM (arm_teleop, arm_teleop_visualizing)
• TeleopProtocol structural interface for future device categories
• Thread-safe: single RLock, monitor-style locking in control loop

Files

• dimos/teleop/ — full subsystem (protocol, quest module, types, extensions, transforms, visualization, blueprints)
• dimos/msgs/std_msgs/UInt32.py — base type for QuestButtons
• dimos/teleop/quest/web/ — Deno bridge + WebXR client

Refer for Detailed Spec

#1113 (Latest Spec at the end)

Linear Stuff

closes DIM-420
closes DIM-394

[greptile-apps]

Greptile Overview

Greptile Summary

This PR adds a complete Meta Quest 3 teleoperation stack spanning:

• WebXR client (dimos/teleop/quest/web/static/index.html) streaming PoseStamped and Joy over WebSocket at ~80Hz.
• Deno bridge (dimos/teleop/quest/web/teleop_server.ts) serving the WebXR UI over HTTPS and forwarding raw LCM packets between browser and UDP LCM.
• Python teleop module (dimos/teleop/quest/quest_teleop_module.py) that subscribes to VR pose/joy topics, transforms WebXR poses into the robot frame (teleop_transforms.py), computes controller deltas (via new Pose.__sub__), and publishes Pose/Twist outputs plus packed button state (QuestButtons).

The change fits into the codebase by registering new module entrypoints and blueprints (dimos/robot/all_blueprints.py, dimos/teleop/blueprints.py) so the teleop pipeline can be instantiated via the existing blueprint/module system.

Main issues to address before merge:

• The Deno server references lcm in the request handler before it is initialized/started, causing runtime failures on first websocket traffic.
• The Python Joy parsing path can raise unhandled ValueError (format mismatch), which can break the subscription callback.
• The module’s conditional subscription logic (if stream.transport) can result in a “starts but never subscribes” behavior depending on how transports are attached.
• QuestButtons bitmask mutation depends on data initialization semantics from the UInt32/LCM base class; ensure data is initialized before bit-ops are used.

Confidence Score: 2/5

• This PR has merge-blocking runtime issues in the Deno bridge and robustness gaps in the Python input handling.
• Score is reduced due to a definite runtime ordering bug in teleop_server.ts (server starts before LCM init), plus unhandled Joy format errors and wiring/subscription behavior that can make the module silently non-functional depending on transport setup.
• dimos/teleop/quest/web/teleop_server.ts, dimos/teleop/quest/quest_teleop_module.py, dimos/teleop/quest/quest_types.py

Important Files Changed

Filename	Overview
.gitignore	Ignores generated teleop TLS cert directory under dimos/assets/teleop_certs/.
dimos/msgs/geometry_msgs/Pose.py	Adds Pose.sub to compute delta pose (position subtraction + quaternion delta).
dimos/msgs/std_msgs/UInt32.py	Adds ROS-compatible UInt32 wrapper; potential base-init concerns for subclasses relying on data being initialized.
dimos/teleop/blueprints.py	Adds arm teleop blueprints wiring Quest teleop outputs to LCM topics.
dimos/teleop/quest/quest_teleop_module.py	Implements core QuestTeleopModule with threads + RLock control loop; issues: uncaught ValueError in _on_joy, RPC/lock layering, and conditional subscription on transport.
dimos/teleop/quest/quest_types.py	Adds QuestControllerState parsing and QuestButtons bitmask; potential initialization/bitmask setattr pitfalls.
dimos/teleop/quest/web/teleop_server.ts	Adds Deno HTTPS+WebSocket to LCM bridge; critical bug: server starts before LCM is initialized, causing runtime failures.
dimos/teleop/utils/teleop_transforms.py	Adds WebXR-to-robot transform utility using matrices + controller-specific Z rotation.

Sequence Diagram

sequenceDiagram
    participant Quest as Quest Browser (WebXR)
    participant Deno as Deno teleop_server.ts
    participant LCM as LCM bus (UDP)
    participant Py as QuestTeleopModule (Python)

    Quest->>Deno: wss:// connect
    loop ~80Hz
        Quest->>Deno: WS binary (LCM packet)<br/>/vr_left_pose, /vr_right_pose
        Quest->>Deno: WS binary (LCM packet)<br/>/vr_left_joy, /vr_right_joy
        Deno->>LCM: publishPacket(packet)
        LCM-->>Py: PoseStamped / Joy
        Py->>Py: _on_pose/_on_joy
    end

    loop 50Hz control loop
        Py->>Py: _handle_engage
        Py->>Py: _get_output_pose (delta)
        Py->>LCM: publish PoseStamped (/teleop/*)
        Py->>LCM: publish QuestButtons (/teleop/buttons)
    end

    LCM-->>Deno: subscribePacket(raw)
    Deno-->>Quest: WS binary (LCM packet)

Loading

[greptile-apps]

_{8 files reviewed, 5 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (5)

dimos/teleop/quest/web/teleop_server.ts
Serve starts before LCM init

Deno.serve(...) is invoked before const lcm = new LCM(); await lcm.start(); (later in the file). The HTTP handler’s websocket path uses await lcm.publishPacket(packet), but lcm hasn’t been initialized yet at server start, so the first websocket upgrade/message will throw (and likely crash the request handler). Initialize/start lcm before calling Deno.serve, or move the server startup below the LCM initialization.

---

dimos/teleop/quest/quest_teleop_module.py
Deadlock via re-entrant lock

_control_loop() holds self._lock for the full iteration, and _handle_engage() calls self.engage()/self.disengage() which are @rpc methods that also acquire self._lock (re-entrant). With threading.RLock this won’t deadlock, but it will run the engage logic twice through two lock layers and can block other threads longer than intended. More importantly, in subclasses overriding _handle_engage() (e.g., ArmTeleopModule), you’re calling engage()/disengage() from inside the lock-held loop too. Consider adding non-RPC internal helpers (e.g., _set_engaged(hand, bool)) that assume the lock is held, and have RPC methods delegate to them, so the control loop doesn’t go through RPC wrappers.

---

dimos/teleop/quest/quest_teleop_module.py
Unhandled ValueError kills callback

QuestControllerState.from_joy() raises ValueError when Joy doesn’t have the expected axis/button lengths, but _on_joy() doesn’t catch it. If the WebXR client sends fewer buttons/axes (common across browsers/firmware), the subscription callback will raise and can terminate/disable that stream. Catch ValueError here (log once/throttle) and ignore the malformed message so the module stays alive.

---

dimos/teleop/quest/quest_teleop_module.py
Inputs silently not subscribed

The if stream and stream.transport guard means the module won’t subscribe unless the In[...] already has a transport attached at start(). In typical blueprint wiring, the transport may be attached after instantiation but before start, but if it’s attached later (or is lazily created), this module will never subscribe and will never receive data. Subscribing unconditionally (and letting .subscribe raise if miswired) or rechecking/attaching on transport changes would prevent a “starts but does nothing” failure mode.

---

dimos/teleop/quest/quest_types.py
Bitmask attr set may break

QuestButtons.__setattr__ mutates self.data via self.data |= ... / &= .... If the underlying LCMUInt32 base class implements data as a property/slot without a pre-existing data attribute at construction, this can raise during from_controllers() when buttons = cls() and then buttons.left_trigger = ... runs. Ensure UInt32.__init__ runs (and sets data) for subclasses, or explicitly initialize data in QuestButtons.__init__ before using bit ops.

[ruthwikdasyam]

All Greptile feedback addressed in latest commits

[paul-nechifor]

dimos/ is only for code. Should this have been assets/teleop_certs/?

[ruthwikdasyam]

Good catch, fixed this
server path + .gitignore are updated to /assets/teleop_certs/

[paul-nechifor]

Since you have the #!/usr/bin/env -S deno run --allow-net --allow-read --allow-run --allow-write --unstable-net she-bang there, couldn't this be just:

./dimos/teleop/quest/web/teleop_server.ts

[ruthwikdasyam]

Yep, updated the README and added the execute bit.
Git tracks the file mode, so it'll be executable for all after this commit.