Merge pull request #337 from open-edge-platform/update-branch

865 user story add ai powered robotic arm demo integration with intel robotics ai suite (#879)

Author: gooishin

usecases/robotic/training-ui/README.md

@@ -0,0 +1,54 @@
+# Robotic Imitation Learning
+
+A lightweight UI and server for robotic imitation learning—covering data capture, conversion, and inference—for a robotic arm demo integrated with Intel® Robotics AI Suite components. The project provides a Python server for data handling and a web UI for interaction.
+
+Robotic imitation learning demo showing a web dashboard controlling a robotic arm during a training session in a lab with UI panels displaying camera view controls, movement controls, and status indicators in a focused technical setting
+
+## Features
+
+- Compatible with Intel® Robotics AI Suite components or Edge Developer Kit Reference Scripts
+- Data capture: Record sensor/telemetry and control events
+- Conversion tools: Optional data format conversion utilities
+- Inference: Run model inference scripts from the server
+- Web UI: Vite-based frontend for control and visualization
+- Modular layout: Separate `server` and `webui` workflows
+
+## Architecture Diagram
+
+![User Flow Diagram](./docs/userflow.jpg)
+![Archictecture Diagram](./docs/archdiagram.jpg)
+
+## Directory Structure
+
+- `server/`: Python server, configs, and scripts
+- `webui/`: Frontend app (Vite + Node.js*)
+
+## Hardware Requirements
+
+- CPU: Intel® Core™ Ultra processors (Series 2) and above
+- GPU: Intel® Arc™ B-Series Graphics (Optional)
+- RAM: 32GB RAM
+- SSD: 500GB and above
+
+## Software Requirements
+
+- **Operating System:** Ubuntu 24.04 LTS
+
+## Setup (Server)
+
+Follow the [doc](server/README.md) to setup
+
+## Setup (Web UI)
+
+Follow the [doc](webui/README.md) to setup
+
+## Trademarks and Notes
+
+- Intel®, Intel® Atom®, Intel® Core™, Intel® Xeon® are trademarks of Intel Corporation.
+- Third-party names such as Ubuntu*, Node.js*, NVM*, Vite*, GitHub* are used for identification purposes only.
+   - Ubuntu* is a trademark of Canonical Limited.
+   - Node.js* and NPM* are trademarks of OpenJS Foundation.
+
+## License
+
+This project is licensed under the Apache License, Version 2.0.

usecases/robotic/training-ui/docs/archdiagram.jpg

@@ -0,0 +1,12 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+data/
+output/

usecases/robotic/training-ui/docs/userflow.jpg

@@ -0,0 +1 @@
+3.11

usecases/robotic/training-ui/server/.gitignore

@@ -0,0 +1,82 @@
+# Server App for Robotic EmbodiedAI
+
+This server application is a core component of the Robotic EmbodiedAI project. It provides backend services, data management, and robot simulation utilities for embodied AI research and development.
+
+## Features
+
+- Robot and teleoperator calibration
+- Data management for simulation and real-world experiments
+- Model training and checkpointing
+- MuJoCo-based simulation runner
+- Modular architecture for easy extension
+
+## Setup
+
+1. **Install system dependencies**
+	```bash
+	sudo apt install build-essential git cmake libevdev-dev python3-dev
+	```
+
+2. **Install UV**
+	```bash
+	curl -LsSf https://astral.sh/uv/install.sh | sh
+	```
+
+3. **Create a Python virtual environment**
+	```bash
+	uv venv --python 3.11 .venv
+	```
+
+
+4. **Install Python dependencies**
+	```bash
+	source .venv/bin/activate
+	uv pip install -r requirements.txt --index-strategy unsafe-best-match
+	uv pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 --index-url https://download.pytorch.org/whl/xpu
+	```
+
+## Run Server
+
+1. **Plug in the devices according to the order**
+	
+	First plug in the leader arm, then followed by the USB/RealSense camera that point at the follower arm. Finally, plug in the follower arm with the camera on the follower arm. Ordering is very important in helping to identify the device later while setting the configurations.
+
+	| Device | Port |
+	| ------ | ---- |
+	| Leader Arm | ttyACM0 |
+	| USB/RealSense Camera | /dev/video0 |
+	| Follower Arm | ttyACM1 |
+	| Camera on Follower Arm | /dev/video2 |
+
+2. **Ensure permission are set for the motor drivers**
+
+	**This must be run whenever you plug out and plug in back the device.** In our case, The leader arm at `TTYACM0` and follower arm at `TTYACM1`.
+	```bash
+	sudo chmod 666 /dev/ttyACM0
+	sudo chmod 666 /dev/ttyACM1
+	```
+
+3. **Calibrate the leader and the follower arm (One time only)**
+	
+	Use the `calibrate-leader.sh` and `calibrate-robot.sh` script to calibrate the leader and the follower arm accordingly. Refer to the video in the [link](https://huggingface.co/docs/lerobot/en/so101#calibration-video) if you are not sure how to do it.
+
+4. **Run the command below to start the server**
+
+	Once started, the server will be brought up at http://localhost:5989.
+	```bash
+	source .venv/bin/activate
+	python main.py
+	```
+
+## Inference with the trained model using OpenVINO
+
+1. Use the script `convert.py` to convert the model to OpenVINO format after training completed. Example command as below. `task-uuid` and `dataset-name` can be refer in the UI.
+	```bash
+	python3 convert.py --model-weight-dir ./output/<task-uuid>/<dataset-name>/checkpoints/last/pretrained_model/ --dataset-dir ./data/datasets/<dataset-name>/
+	```
+
+2. Use the provided `inference.sh` script to run the inference easily. Configure the `CAMERA_CONFIG` and `POLICY_PATH` based on your training dataset. Configure the `OPENVINO_DEVICE` to run inference on different devices on Intel platform.
+
+## License
+
+See the root project for license information.

usecases/robotic/training-ui/server/.python-version

@@ -0,0 +1,70 @@
+# Copyright (C) 2025 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+import zmq
+import time
+import gymnasium as gym
+import numpy as np
+import json
+import gym_hil  # noqa: F401
+
+context = zmq.Context()
+socket = context.socket(zmq.PUB)
+socket.bind("tcp://*:5556")
+
+def main():
+    env_id = "gym_hil/PandaPickCubeKeyboard-v0"
+    env = gym.make(
+        env_id,
+        render_mode="human",
+        image_obs=True,
+        use_gamepad=False,
+        max_episode_steps=1000,  # 100 seconds * 10Hz
+    )
+
+    # Print observation space for the wrapped environment
+    print("Wrapped observation space:", env.observation_space)
+
+    # Reset and check wrapped observation structure
+    obs, _ = env.reset()
+    print("Wrapped observation keys:", list(obs.keys()))
+
+    # Reset environment
+    obs, _ = env.reset()
+    dummy_action = np.zeros(4, dtype=np.float32)
+    dummy_action[-1] = 1
+
+    topic = "live_data"
+
+    try:
+        while True:
+            # Step the environment
+            obs, reward, terminated, truncated, info = env.step(dummy_action)
+            
+            try:
+                socket.send_string("NUMPY_DATA", zmq.SNDMORE)
+                socket.send_pyobj(obs)
+                # socket.send_multipart([topic.encode('utf-8'), message.encode('utf-8')])
+            except:
+                pass
+            
+            # Print some feedback
+            if info.get("succeed", False):
+                print("\nSuccess! Block has been picked up.")
+
+            # If auto-reset is disabled, manually reset when episode ends
+            if terminated or truncated:
+                print("Episode ended, resetting environment")
+                obs, _ = env.reset()
+
+            time.sleep(0.1)
+
+    except KeyboardInterrupt:
+        print("Interrupted by user")
+    finally:
+        env.close()
+        print("Session ended")
+
+
+if __name__ == "__main__":
+    main()

usecases/robotic/training-ui/server/README.md

@@ -0,0 +1,56 @@
+# Copyright (C) 2025 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+import zmq
+import sys
+import numpy as np
+import cv2
+
+# Define the connection details
+ZMQ_ADDRESS = "tcp://10.158.108.112:5556"
+# The topic we want to subscribe to (e.g., 'STATUS'). 
+# Use b'' or b' ' for all topics.
+SUBSCRIPTION_TOPIC = b"live_data" 
+
+# 1. Get the ZeroMQ context
+context = zmq.Context()
+
+# 2. Create the SUB socket
+socket = context.socket(zmq.SUB)
+
+# 3. Connect to the publisher server
+print(f"Connecting to ZMQ publisher at {ZMQ_ADDRESS}...")
+socket.connect(ZMQ_ADDRESS)
+
+# 4. Set the subscription filter
+# The socket receives only messages starting with this topic byte string.
+# An empty string (b'') subscribes to ALL messages.
+socket.subscribe("NUMPY_DATA")
+print(f"Subscribed to topic: {SUBSCRIPTION_TOPIC.decode()}")
+
+try:
+    # 5. Start the message receiving loop
+    while True:
+        # zmq.SUB sockets often send messages in two parts (multipart):
+        # [Topic, Message]
+        topic = socket.recv_string() 
+
+        # Use recv_multipart() to get both parts
+        received_array = socket.recv_pyobj() 
+        
+        for k, v in received_array['pixels'].items():
+            if isinstance(v, np.ndarray):
+                cv2.imshow("frame", v)
+        
+        cv2.waitKey(10)
+
+        # print(f"Received topic: {topic}")
+        # print(f"Received array:\n{received_array}")
+
+
+except KeyboardInterrupt:
+    print("\nShutting down SUB client...")
+finally:
+    # 6. Clean up
+    socket.close()
+    context.term()

usecases/robotic/training-ui/server/apps/mujoco_runner.py

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2025 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+set -euo pipefail
+
+lerobot-calibrate --teleop.type=so101_leader --teleop.port=/dev/ttyACM0 --teleop.id=my_leader_arm

usecases/robotic/training-ui/server/apps/sub.py

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2025 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+set -euo pipefail
+
+lerobot-calibrate --robot.type=so101_follower --robot.port=/dev/ttyACM1 --robot.id=my_follower_arm