ArcaneHand: Hand Pose Estimation from Apple Watch

Transform Apple Watch IMU data into realistic 3D hand animations using deep learning

🚀 Overview

This project converts IMU sensor data from Apple Watch into realistic 3D hand mesh animations using the MANO hand model. Our deep learning approach captures natural hand movements and finger articulation from wrist-worn inertial sensors.

How It Works

1. 📱 Data Collection: Apple Watch records accelerometer and gyroscope data
2. 🧠 Neural Network: Enhanced LSTM processes 7 IMU features (acc_x, acc_y, acc_z, rot_w, rot_x, rot_y, rot_z)
3. ✋ Hand Generation: Outputs MANO parameters for realistic 3D hand mesh
4. 🎬 Animation: Creates smooth hand animations with quantile-based spike removal

Key Features

• Real-time inference from Apple Watch IMU data
• Per-finger accuracy with dedicated joint angle losses
• Quantile-based smoothing for natural hand movements
• Both hands supported (left/right hand detection)
• Professional video output with multiple camera angles

📊 Dataset

Our model is trained on 2+ hours of hand gesture data collected from 30+ participants using Apple Watch, ensuring robust generalization across different users and hand movements.

🎯 Pre-trained Model

Download our pre-trained model: 📥 Pre-trained Weights

The model achieves state-of-the-art performance on IMU-to-hand pose estimation with enhanced finger articulation accuracy.

🛠️ Quick Setup

1. Installation

# Clone repository
git clone https://github.com/yourusername/ArcaneHand.git
cd ArcaneHand

# Install dependencies
pip install torch torchvision numpy pandas scipy matplotlib imageio tqdm
pip install smplx  # For MANO hand model

# Download MANO model (required)
# Get MANO_RIGHT.pkl from: https://mano.is.tue.mpg.de/
# Place in project root directory

2. Download Pre-trained Model

Download best_enhanced_finger_model.pth and place it in the project root directory.

3. Prepare Your Data

Your IMU data should be in CSV format with the following structure:

time,acc_x,acc_y,acc_z,rot_w,rot_x,rot_y,rot_z
0.000,-0.123,0.456,9.789,0.999,0.001,-0.002,0.003
0.020,-0.125,0.458,9.791,0.998,0.002,-0.001,0.004
0.040,-0.127,0.460,9.793,0.997,0.003,0.000,0.005
...

Required columns:

• time: Timestamp (optional, for reference)
• acc_x, acc_y, acc_z: Linear acceleration in m/s²
• rot_w, rot_x, rot_y, rot_z: Quaternion components (normalized)

🚀 Usage

Simple Inference

python inference.py --input data/my_gesture.csv --output results/ --hand right

With Custom Settings

python inference.py \
    --input data/my_gesture.csv \
    --output results/ \
    --hand right \
    --smooth \
    --video \
    --fps 30

Python API

from inference import IMUToHandPredictor

# Initialize predictor
predictor = IMUToHandPredictor(
    model_path='best_enhanced_finger_model.pth',
    mano_path='MANO_RIGHT.pkl'
)

# Load and process your data
results = predictor.predict_from_csv(
    'data/my_gesture.csv',
    hand_type='right',
    apply_smoothing=True
)

# Generate video
predictor.create_video(
    results['poses'],
    'output/hand_animation.mp4',
    fps=30
)

📁 Project Structure

ArcaneHand/
├── README.md                    # This file
├── inference.py                 # Main inference script 🔥
├── model_utils.py              # Model utilities 🔥
├── requirements.txt            # Dependencies
├── best_enhanced_finger_model.pth  # Pre-trained model (download)
├── MANO_RIGHT.pkl              # MANO model (download)
├── data/                       # Your IMU data
│   ├── example_gesture.csv     # Example data format
│   └── my_gesture.csv          # Your data
├── results/                    # Output directory
│   ├── poses/                  # Generated poses
│   ├── meshes/                 # 3D meshes
│   └── videos/                 # Generated videos
└── examples/                   # Example scripts
    ├── basic_usage.py
    └── advanced_usage.py

📋 Data Format Requirements

Input CSV Format

Your CSV file must contain exactly these 7 columns (in any order):

Column	Description	Units	Range
acc_x	X-axis acceleration	m/s²	-50 to +50
acc_y	Y-axis acceleration	m/s²	-50 to +50
acc_z	Z-axis acceleration	m/s²	-50 to +50
rot_w	Quaternion W component	unitless	-1 to +1
rot_x	Quaternion X component	unitless	-1 to +1
rot_y	Quaternion Y component	unitless	-1 to +1
rot_z	Quaternion Z component	unitless	-1 to +1

Example Data File

time,acc_x,acc_y,acc_z,rot_w,rot_x,rot_y,rot_z
0.000,-0.123,0.456,9.789,0.999,0.001,-0.002,0.003
0.020,-0.125,0.458,9.791,0.998,0.002,-0.001,0.004
0.040,-0.127,0.460,9.793,0.997,0.003,0.000,0.005
0.060,-0.129,0.462,9.795,0.996,0.004,0.001,0.006
0.080,-0.131,0.464,9.797,0.995,0.005,0.002,0.007

Data Preprocessing (Important!)

If your data comes from Apple Watch or other sources, you may need to:

1. Unit Conversion: Convert accelerometer from g-force to m/s² (multiply by 9.80665)
2. Quaternion Normalization: Ensure quaternions have unit magnitude
3. Sampling Rate: Resample to 30-50 Hz for best results
4. Filtering: Apply gentle low-pass filter (optional)

# Example preprocessing
import pandas as pd
import numpy as np

def preprocess_apple_watch_data(csv_file):
    df = pd.read_csv(csv_file)
    
    # Convert g to m/s² (if needed)
    if df['acc_x'].abs().max() < 20:  # Likely in g units
        df['acc_x'] *= 9.80665
        df['acc_y'] *= 9.80665
        df['acc_z'] *= 9.80665
    
    # Normalize quaternions
    quat_cols = ['rot_w', 'rot_x', 'rot_y', 'rot_z']
    quat_mag = np.sqrt((df[quat_cols] ** 2).sum(axis=1))
    df[quat_cols] = df[quat_cols].div(quat_mag, axis=0)
    
    return df

🎬 Output Formats

The inference script generates multiple outputs:

1. Pose Parameters

• File: results/poses/gesture_poses.npy
• Format: NumPy array [frames, 48] containing MANO parameters

2. 3D Mesh Vertices

• File: results/meshes/gesture_vertices.npy
• Format: NumPy array [frames, 778, 3] containing 3D hand vertices

3. Video Animation

• File: results/videos/gesture_animation.mp4
• Format: MP4 video showing 3D hand animation

4. Processing Report

• File: results/processing_report.json
• Contains: Processing statistics, smoothing info, quality metrics

🎛️ Command Line Options

python inference.py [OPTIONS]

Required:
  --input PATH          Input CSV file with IMU data
  --output PATH         Output directory for results

Optional:
  --hand {left,right}   Hand type (default: right)
  --model PATH          Path to model file (default: best_enhanced_finger_model.pth)
  --mano PATH           Path to MANO model (default: MANO_RIGHT.pkl)
  --smooth              Apply quantile-based smoothing
  --video               Generate video output
  --fps INT             Video frame rate (default: 30)
  --format {mp4,gif}    Video format (default: mp4)
  --verbose             Verbose output

🔧 Advanced Usage

Batch Processing

from inference import IMUToHandPredictor
import glob

predictor = IMUToHandPredictor()

# Process multiple files
csv_files = glob.glob('data/*.csv')
for csv_file in csv_files:
    results = predictor.predict_from_csv(csv_file)
    predictor.save_results(results, f'results/{Path(csv_file).stem}/')

Custom Model Configuration

# Load with custom settings
predictor = IMUToHandPredictor(
    model_path='custom_model.pth',
    device='cuda',
    smoothing_config={
        'spike_percentile': 85,  # Remove top 15% spikes
        'window_size': 7,
        'min_threshold': 0.005
    }
)

📊 Performance Tips

• Sampling Rate: 30-50 Hz works best
• Sequence Length: 100-1000 frames optimal
• Data Quality: Clean data = better results
• Smoothing: Enable for noisy sensor data
• GPU: Use CUDA for faster processing

🤝 Contributing

We welcome contributions! Please see our Contributing Guide.

📝 Citation

@article{ArcaneHand_2025,
  title={Real-time Hand Pose Estimation from Apple Watch IMU Data},
  author={Faraz Rabbani},
  year={2025}
}

📄 License

MIT License - see LICENSE file.

---