RTMlib Interactive Effects

Real-time interactive visual effects driven by human movement, powered by RTMlib pose estimation and GPU-accelerated simulation.

Overview

This project combines real-time pose estimation with dynamic visual effects overlaid on a webcam feed. Two systems are available:

• Fluid Smoke & Fireball (particle_game/) — the primary project. Eulerian fluid simulation produces organic smoke trails rising from fingertips. Flick a finger to launch a fireball projectile that leaves a fire-colored trail transitioning to smoke. Modular architecture with clean separation of concerns.
• Particle Effects (scripts/particle_effects_CUDA.py) — the original monolithic script. Discrete particle trails, fireball shooting, monster sprites, and sound effects.

---

Fluid Smoke (particle_game/)

Smooth, organic fluid smoke trailing from fingertips, overlaid transparently on the webcam feed. The smoke rises, spreads, and fades naturally using a Stable Fluids solver running entirely on GPU via NVIDIA Warp. In fireball mode, a fast straight fingertip flick launches a projectile that leaves a fire-colored trail (white core → orange → dark red) that gradually transitions to smoke as the heat dissipates.

How It Works

flowchart LR
    A["Webcam (Capture)"] --> B["RTMPose (Detection)"]
    B --> C["Warp GPU (Fluid Sim)"]
    C --> D["OpenGL (Density+Heat Render)"]
    D --> E["OpenCV (Composite)"]

    style C fill:#f96,stroke:#333,stroke-width:2px
    style D fill:#f96,stroke:#333,stroke-width:2px

Loading

1. Threaded Capture: Webcam frames grabbed continuously in a background thread (zero blocking)
2. Pose Detection: RTMlib detects keypoints — default: left & right index fingertips
3. Keypoint Smoothing: One Euro Filter removes jitter at rest while preserving fast gesture fidelity
4. Fluid Injection: Density and upward velocity injected at fingertip positions into a 640x480 grid. Injected density is velocity-dependent — full density when stationary, thinning to a wispy trail during fast gestures. Line-segment stepping handles fast movement between frames. Keypoints also push existing smoke aside by injecting movement velocity into the fluid (disturbance). In fireball mode, projectiles inject high-density fluid with heat=1.0 along their path.
5. Fluid Simulation: Stable Fluids solver (NVIDIA Warp GPU kernels) — advection of velocity, density, and heat; pressure projection (100 Jacobi iterations via CUDA graph); buoyancy; density and heat fade (heat fades faster so fire trails transition to smoke)
6. Density+Heat Rendering: OpenGL fragment shader samples both density and heat textures. Density controls alpha. Heat blends between smoke color ramp (grey → white) and fire color ramp (dark red → orange → near-white). Areas without smoke are fully transparent.
7. Compositing: Integer alpha blend (uint16, zero allocation) overlays smoke onto webcam frame
8. Display: Mirror-flip + 2x upscale in a single cv2.warpAffine call to 1280x960 display

Usage

cd particle_game
conda run -n rtmlib python main.py

ONNX models are downloaded automatically on first run.

Controls

Key	Action
q	Quit
o	Toggle timing overlay
e	Toggle fluid effect on/off
k	Toggle keypoint debug circles
f	Toggle smoke / fireball mode

Gestures (fireball mode)

Gesture	Effect
Fast straight fingertip flick	Fires a fireball projectile that leaves a fire-colored trail

Architecture

particle_game/
├── config.py           # All constants (tracking, fluid, heat, fireball, rendering, display)
├── capture.py          # CaptureThread (threaded webcam capture)
├── pose.py             # PoseDetector, OneEuroFilter, velocity helpers
├── effects/
│   ├── base.py         # Abstract Effect interface
│   └── fluid.py        # Eulerian fluid simulation (Warp GPU kernels, density + heat)
├── renderer.py         # OpenGL density+heat texture rendering (dual color ramp)
├── projectile.py       # Projectile, ProjectileManager, shoot detection
├── compositing.py      # uint16 alpha composite + flip/scale
└── main.py             # Game loop orchestration

Configuration

Fluid parameters in particle_game/config.py:

# Fluid simulation
FLUID_SUBSTEPS = 2
PRESSURE_ITERATIONS = 100
FLUID_GRAVITY = (0.0, -120.0)  # Buoyancy (upward)
DENSITY_FADE_RATE = 0.1        # Exponential decay per second
INJECT_RADIUS = 4.0            # Injection radius in pixels
INJECT_VELOCITY = (0.0, -50.0) # Fixed upward velocity

# Velocity-dependent density (thinner trails when moving fast)
VELOCITY_DENSITY_MAX_SPEED = 20.0  # px/frame at which density bottoms out
VELOCITY_DENSITY_MIN = 0.2         # Minimum density scale (at max speed)

# Heat field (passive scalar for fire rendering)
HEAT_FADE_RATE = 0.3           # Faster than density — fire → smoke transition

# Fireball injection (projectile trail)
FIREBALL_INJECT_DENSITY = 4.0  # Higher density for fire
FIREBALL_INJECT_HEAT = 1.0     # Full heat
FIREBALL_INJECT_RADIUS = 3.0   # Tighter than smoke

# Shoot detection
SHOOT_DT = 3                   # Frames to look back
SHOOT_DISTANCE_THRESHOLD = 50  # Min displacement (px)
SHOOT_LINEARITY_THRESHOLD = 10 # Max perpendicular deviation (px)

# Smoke color ramp (RGB 0.0-1.0, low → high density)
COLOR_RAMP_LOW  = (0.1, 0.1, 0.1)     # Dark grey
COLOR_RAMP_MID  = (0.55, 0.55, 0.55)  # Mid grey
COLOR_RAMP_HIGH = (1.0, 0.95, 0.95)   # Near white

# Fire color ramp (blended in by heat field)
FIRE_COLOR_LOW  = (0.3, 0.05, 0.0)    # Dark red
FIRE_COLOR_MID  = (1.0, 0.4, 0.0)     # Orange
FIRE_COLOR_HIGH = (1.0, 0.9, 0.6)     # Near-white/yellow

Tracking Modes

Set TRACKING_MODE in config.py:

Mode	Model	Keypoints	Default emission
"wholebody" (default)	YOLOX-tiny + RTMPose DW-L-M	133 (body + face + hands)	Ears emit, left index emit+disturb, right index disturb
"body"	RTMO	17 (COCO17)	Left & right wrists (emit)

Each keypoint in KEYPOINT_SELECTION has a role bitmask: 0 = off, 1 = emit (creates smoke), 2 = disturb (pushes existing smoke), 3 = both.

---

Legacy: Particle Effects (scripts/particle_effects_CUDA.py)

The original monolithic script with discrete GPU-accelerated particle effects and game mechanics.

Features

• GPU particle physics via NVIDIA Warp with instanced OpenGL rendering
• Metaball rendering — two-pass energy-field rendering with blobby merging shapes (m key)
• Fireball shooting — fast straight fingertip flick fires a projectile (f key)
• Monster sprites — spawn at edges, creep toward nose, 3 sprite states (normal, surprised, dead)
• Mouth-open attraction — monsters rush to mouth when held open
• Ear spit effect — swallowed monsters shot back from ears on mouth close
• Pinch explosion — hold index tips together for radial blast that kills all monsters
• Sound effects — synthesized audio for every interaction
• Keypoint smoothing with One Euro Filter
• Path interpolation — pchip, akima for smooth emission trails

Usage

conda run -n rtmlib python scripts/particle_effects_CUDA.py

Controls

Key	Action
p	Toggle particle effects
f	Toggle smoke / fireball mode
m	Toggle metaball / classic rendering
k	Toggle keypoint circles
o	Toggle timing overlay
s	Toggle sound
b	Toggle bounding boxes (wholebody mode)
q	Quit

Gestures (wholebody mode)

Gesture	Effect
Fast straight fingertip flick	Fires a fireball projectile
Hold both index tips together ~1.5 s	Pinch explosion — kills all monsters
Hold mouth wide open ~1.5 s	Monsters accelerate toward mouth, swallowed on contact
Close mouth (after attraction)	Swallowed monsters shot back from ears

---

Requirements

• NVIDIA GPU with up-to-date drivers (CUDA Toolkit installation is not needed — warp-lang and onnxruntime-gpu bundle their own CUDA runtimes)
• Python 3.10+
• ONNX Runtime GPU (onnxruntime-gpu) — runs RTMPose inference via CUDAExecutionProvider
• NVIDIA Warp (warp-lang) — runs fluid simulation kernels on CUDA
• OpenCV (opencv-python), NumPy, PyOpenGL, PyOpenGL-accelerate, Pygame
• RTMlib
• Legacy script additionally needs: SciPy, Pillow

Installation

1. Set up a conda environment:

   conda create -n rtmlib python=3.10
   conda activate rtmlib
   

2. Install RTMlib:

   pip install rtmlib -i https://pypi.org/simple
   

3. Install dependencies:

   pip install onnxruntime-gpu warp-lang opencv-python numpy pygame pyopengl pyopengl-accelerate pillow scipy
   

---

Acknowledgements

• RTMlib: Lightweight library for real-time pose estimation
• RTMPose: Real-Time Multi-Person Pose Estimation
• MMPose: OpenMMLab Pose Estimation Toolbox
• NVIDIA Warp: Python framework for GPU-accelerated simulation
• PyOpenGL: Python bindings for OpenGL

License

This project is released under the same license as RTMlib.