Skip to content

Commit 3625d78

Browse files
jcgglclaude
andcommitted
feat: replace simplified examples with actual production demo code
Port real working demo code from lipsync-wasm v1/test, v2/test, and demo/ into AnimaSync examples. All imports converted from local paths to jsdelivr CDN (@0.3.9). Example READMEs and landing page updated to reflect V1 (111-dim phoneme) and V2 (52-dim student model) demos. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 8bcc0d9 commit 3625d78

File tree

7 files changed

+4553
-1382
lines changed

7 files changed

+4553
-1382
lines changed

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -130,8 +130,8 @@ Working examples you can run locally — zero npm install, all loaded from CDN.
130130

131131
| Example | Description | Source |
132132
|---------|-------------|--------|
133-
| **[Basic](examples/vanilla-basic/)** | Audio → animated blendshape visualization. No 3D, pure API demo. | [index.html](examples/vanilla-basic/index.html) |
134-
| **[VRM Avatar](examples/vanilla-avatar/)** | Full 3D avatar — lip sync, expressions, body motion, mic streaming. | [index.html](examples/vanilla-avatar/index.html) |
133+
| **[V1 Avatar](examples/vanilla-basic/)** | Full 3D VRM avatar with V1 (111-dim phoneme engine). Dual mode, VAD, idle expressions. | [index.html](examples/vanilla-basic/index.html) |
134+
| **[V2 Avatar](examples/vanilla-avatar/)** | Full 3D VRM avatar with V2 (52-dim student model). Crisp mouth, direct prediction. | [index.html](examples/vanilla-avatar/index.html) |
135135
| **[V1 vs V2](examples/vanilla-comparison/)** | Side-by-side dual avatar comparison. Same voice, two animation engines. | [index.html](examples/vanilla-comparison/index.html) |
136136

137137
**Run any example:**

examples/vanilla-avatar/README.md

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,16 @@
1-
# Vanilla Avatar
1+
# Vanilla Avatar (V2)
22

3-
Full 3D VRM avatar that comes alive from voice alone. Lip sync, emotional facial expressions, natural eye blinks, and body motion — all generated from a single audio stream via AnimaSync V2.
3+
Full 3D VRM avatar driven by AnimaSync V2 — the 52-dim student model engine. Lip sync, facial expressions, natural eye blinks, and body motion — all generated from a single audio stream via direct blendshape prediction.
44

55
## What it demonstrates
66

7-
- **Lip sync**: Mouth shapes driven by voice phonemes
8-
- **Facial expressions**: Brows, cheeks, and eye area respond to vocal characteristics
9-
- **Eye animation**: Natural stochastic blinks injected automatically
7+
- **52-dim ARKit output**: Standard blendshape channels via student model direct prediction
8+
- **Lip sync**: Crisp mouth shapes with threshold-based sharpening
9+
- **Facial expressions**: Brows and eye area respond to vocal characteristics
10+
- **Eye animation**: Natural stochastic blinks injected by post-processing
1011
- **Body motion**: VRMA bone animation (idle breathing ↔ speaking pose crossfade)
11-
- Real-time mic streaming + batch file processing
12+
- **Post-processing**: crisp_mouth + fade_in_out + add_blinks pipeline
13+
- Real-time mic streaming + batch file processing + TTS
1214
- Three.js + `@pixiv/three-vrm` integration
1315

1416
## Run locally
@@ -24,11 +26,10 @@ python3 -m http.server 8080
2426
Drop any `.vrm` file onto the canvas. Free CC0 avatars are available at:
2527

2628
- [VRoid Hub](https://hub.vroid.com/en/models?characterization=allow) — filter by "OK to use as-is"
27-
- [Mixamo](https://www.mixamo.com/) — for reference animations
2829

2930
## How it works
3031

3132
1. Page loads → WASM + ONNX model initialized from CDN
3233
2. Drop a `.vrm` file → Three.js scene renders the avatar with idle breathing animation
33-
3. Upload audio or click Microphone → engine generates lip sync + expressions + blinks
34+
3. Upload audio or click Microphone → V2 engine generates direct lip sync + expressions + blinks
3435
4. All animation layers (face + body) applied to VRM at 30fps via frame queue

examples/vanilla-avatar/index.html

Lines changed: 1316 additions & 487 deletions
Large diffs are not rendered by default.

examples/vanilla-basic/README.md

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,36 @@
1-
# Vanilla Basic
1+
# Vanilla Basic (V1)
22

3-
Minimal AnimaSync example — no 3D avatar, no Three.js. Drop an audio file and see how voice drives lip sync, facial expression, and blink animation data in real time.
3+
Full 3D VRM avatar driven by AnimaSync V1 — the 111-dim phoneme-based engine. Lip sync, facial expressions (brows, cheeks, tongue), natural eye blinks, and body motion generated from voice via ONNX phoneme classification + viseme mapping.
44

55
## What it demonstrates
66

7-
- Loading `@goodganglabs/lipsync-wasm-v2` from CDN (zero `npm install`)
8-
- `processFile()` batch API — returns lip sync + expressions + blinks in one call
9-
- Visualizing 23 key ARKit channels: jaw, mouth, eyes, brows, cheeks
7+
- **111-dim blendshape output**: Full ARKit channels including tongue, cheeks, and brows
8+
- **Phoneme-based pipeline**: Voice → MFCC → Phoneme → Viseme → Blendshape
9+
- **Dual mode**: ONNX inference with heuristic fallback if ONNX fails
10+
- **IdleExpressionGenerator**: Natural eye blinks (2.5–4.5s cycle, double-blink 15%)
11+
- **VoiceActivityDetector**: Auto-switches idle ↔ speaking body pose
12+
- **OneEuroFilter**: Time-domain smoothing for natural motion
13+
- Real-time mic streaming + batch file processing + TTS
14+
- Three.js + `@pixiv/three-vrm` integration
1015

1116
## Run locally
1217

1318
```bash
14-
# Any static file server works
1519
npx serve .
1620
# or
1721
python3 -m http.server 8080
1822
```
1923

20-
Open `http://localhost:8080` (or the port your server shows).
24+
## VRM Avatar
2125

22-
## How it works
26+
Drop any `.vrm` file onto the canvas. Free CC0 avatars are available at:
27+
28+
- [VRoid Hub](https://hub.vroid.com/en/models?characterization=allow) — filter by "OK to use as-is"
2329

24-
1. WASM + ONNX model load from jsdelivr CDN on page load
25-
2. Drop/select an audio file → `processFile()` returns all animation frames (lip sync + expressions + blinks)
26-
3. `requestAnimationFrame` loop plays frames at 30fps, showing how each facial channel responds to the voice
30+
## How it works
2731

28-
No bundler, no framework, single HTML file.
32+
1. Page loads → WASM + ONNX model initialized from CDN
33+
2. Drop a `.vrm` file → Three.js scene renders the avatar with idle breathing animation
34+
3. Upload audio or click Microphone → V1 engine generates phoneme-based lip sync + expressions + blinks
35+
4. All animation layers (face + body) applied to VRM at 30fps via frame queue
36+
5. Body pose auto-transitions between idle and speaking based on voice activity detection

0 commit comments

Comments
 (0)