Live mode: video pipeline, GPU bridge, UX controls, DB isolation#276
Merged
Live mode: video pipeline, GPU bridge, UX controls, DB isolation#276
Conversation
…n fix Phase 1 — Stabilize: - AVATAR_FPS 24→15 (~40% GPU/CPU reduction for 16 concurrent avatars) - Wrap spawn_renderer_loop in spawn_blocking (std::sync::Mutex was starving tokio) - Accumulate 160-sample LiveKit frames to 480-sample chunks for earshot VAD (VAD requires ≥240 samples; was always returning silence) - VideoPublishState struct, max_bitrate 1.2M→800k, max_framerate 24→15 Phase 2 — FramePublisher abstraction: - FramePublisher trait: try_publish() decouples video loop from format conversion - CpuI420Publisher: RGBA→I420 (BT.601), works on all platforms (default) - create_publisher() factory selects best publisher per platform - rgba_to_i420_into() moved to frame_publisher.rs (single source of truth) - Unit tests for BT.601 conversion correctness Phase 3 — NativeBufferPublisher (macOS, disabled by default): - CVPixelBuffer BGRA path via CoreVideo FFI (skips I420 conversion) - Enabled via CONTINUUM_NATIVE_VIDEO=1 for testing - Dark frames issue: WebRTC encoder doesn't handle BGRA CVPixelBuffers correctly - Falls back to CpuI420Publisher on failure or non-macOS
Pre-allocate double-buffered IOSurface-backed NV12 CVPixelBuffers per slot. Bevy ReadbackComplete observer writes RGBA→NV12 directly to IOSurface via atomic frame counter — eliminates pixel_bytes.to_vec() (1.2MB) and per-frame CVPixelBufferCreate (460KB). Publisher wraps IOSurface in lightweight CVPixelBuffer metadata (~40 bytes) via CVPixelBufferCreateWithIOSurface. At 195 frames/sec (13 agents × 15fps): ~324 MB/s alloc → ~7.6 KB/s. Three-tier macOS cascade: GpuBridge → NativeBuffer → CpuI420. CONTINUUM_CPU_VIDEO=1 escape hatch preserved.
Align Rust module naming with TypeScript's existing live/ convention. The voice/ module contained audio, video, avatars, transport — the entire live experience, not just voice. Now properly organized: live/ ├── audio/ (buffer, router, mixer, stt/, tts/, vad/) ├── avatar/ (catalog, selection, publishers, render_loop) ├── video/ (bevy_renderer, generator, source) ├── transport/ (livekit_agent, call_server, media) └── session/ (orchestrator, voice_service, tests) Module declaration order in live/mod.rs is load-bearing: webrtc-dependent modules (avatar, video, transport) must be declared before ort-dependent modules (audio) to avoid protobuf symbol conflicts between libwebrtc.a and libonnxruntime.a which bundle different protobuf C++ versions. Also fixes executable permissions on all scripts/*.sh files.
… strip and controls Captions were absolute-positioned at a fixed offset (bottom: 56px) which overlapped the participant strip in spotlight mode and bottom grid rows in grid mode. Then the flex-flow fix put them as a separate bar between strip and controls — wrong place. Now: wrap main content (grid or spotlight-main) in .content-area with position:relative. Captions are position:absolute at bottom of that area, overlaying the video content like Zoom/Teams/Meet do it. Strip and controls are outside the overlay zone.
- Add captionsEnabled to CallState type and default (UserStateEntity) - Load/save captionsEnabled in LiveWidget state persistence - Replace abstract SVG caption icon with universal "CC" text in box (matches Zoom, Meet, Teams standard)
- LiveParticipantTile: isPinned prop, hover overlay bar with name + pushpin SVG icon, pin-participant custom event, pinned CSS class with cyan border - LiveWidget: pin-participant event wiring, _onPinParticipant toggle handler, isPinned passed to all tiles in grid and spotlight views - Tile click = temporary spotlight (auto-speaker can override), pin icon click = locked pin (only way to pin) - LiveCaptions: always render DOM, toggle display via CSS (no conditional DOM insertion/removal) - toggleCaptions now calls saveCallState() (CC preference persists) - _stateLoaded guard prevents controls rendering with wrong defaults - SCSS: z-index layering (video:1, name:3, overlay:5), prominent hover overlay with gradient bg, 32px pin button
- BaseUser.personalDbHandle: every citizen gets personal storage - Hippocampus sets handle on parent directly (no propagation step) - PersonaMemory reads handle via live getter reference, requireHandle() throws if null - LimbicSystem.ensureDbReady() replaces propagateDbHandle(), throws on failure - PersonaUser init order: hippocampus → ensureDbReady → genome → logger → autonomous loop (last) - Hippocampus throws on DB open failure (no silent STM-only fallback) - CognitionLogger: cleaned unused imports, init order guarantees handle populated
Contributor
There was a problem hiding this comment.
Pull request overview
This PR migrates the “voice” subsystem into a broader live experience module (voice + video + transport), adds GPU/video pipeline pieces (including adaptive resolution hooks), updates the Live widget UX (captions/pin/CC), and isolates per-persona persistent storage via per-user DB handles.
Changes:
- Rename/restructure
voice/→live/across Rust modules/tests and update runtime/module wiring. - Add/extend live video + avatar rendering pipeline (slot resize, GPU bridge hooks) and VAD/STT/TTS plumbing.
- Update Live widget layout + controls (caption overlay placement, persisted CC state, pin UX) and persona DB isolation/init order.
Reviewed changes
Copilot reviewed 67 out of 115 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/workers/continuum-core/tests/voice_routing_integration.rs | Update test imports to continuum_core::live |
| src/workers/continuum-core/tests/stt_noise_benchmark.rs | Update VAD test imports to new live/audio paths |
| src/workers/continuum-core/tests/ipc_voice_tests.rs | Update IPC voice test imports to live |
| src/workers/continuum-core/tests/hold_music_test.rs | Update CallManager import path under live/transport |
| src/workers/continuum-core/tests/call_server_routing_test.rs | Update CallManager import path under live/transport |
| src/workers/continuum-core/tests/call_server_integration.rs | Update imports + UtteranceEvent path to live |
| src/workers/continuum-core/src/voice/mod.rs | Remove old voice module root exports |
| src/workers/continuum-core/src/runtime/runtime.rs | Expect live module name; minor logging brace change |
| src/workers/continuum-core/src/modules/mod.rs | Register live module instead of voice |
| src/workers/continuum-core/src/modules/live.rs | Port VoiceModule wiring to crate::live::* namespaces |
| src/workers/continuum-core/src/main.rs | Update STT/TTS init paths and LiveKit manager import |
| src/workers/continuum-core/src/logging/mod.rs | Map live::* paths to modules/live log category |
| src/workers/continuum-core/src/live/video/source.rs | Update handle/generator imports to live paths |
| src/workers/continuum-core/src/live/video/mod.rs | Declare live video submodules |
| src/workers/continuum-core/src/live/video/generator.rs | Update VideoFrame type imports to live::types |
| src/workers/continuum-core/src/live/video/bevy_renderer.rs | Adaptive slot sizing + GPU bridge hook + updated defaults |
| src/workers/continuum-core/src/live/types.rs | Add HD/FullHD tiers to wire enum |
| src/workers/continuum-core/src/live/transport/mod.rs | Declare live transport submodules |
| src/workers/continuum-core/src/live/transport/media.rs | Update Handle import to live::handle |
| src/workers/continuum-core/src/live/transport/call_server.rs | Port call server to live/* modules; fix hold music include path |
| src/workers/continuum-core/src/live/session/voice_service.rs | Update orchestrator/type imports to crate::live |
| src/workers/continuum-core/src/live/session/tests/orchestrator_tests.rs | Update test module imports |
| src/workers/continuum-core/src/live/session/tests/mod.rs | Add session tests module entry |
| src/workers/continuum-core/src/live/session/tests/call_server_orchestrator_test.rs | Update test imports to crate::live |
| src/workers/continuum-core/src/live/session/orchestrator.rs | Update module paths + fix test path include |
| src/workers/continuum-core/src/live/session/mod.rs | Declare session submodules + tests |
| src/workers/continuum-core/src/live/mod.rs | New live module root + re-exports + link-order comment |
| src/workers/continuum-core/src/live/handle.rs | New universal Handle type |
| src/workers/continuum-core/src/live/avatar/types.rs | New avatar model/type definitions + TS exports |
| src/workers/continuum-core/src/live/avatar/renderer.rs | New AvatarRenderer trait |
| src/workers/continuum-core/src/live/avatar/render_loop.rs | Slot pooling + RAII SlotGuard + direct Bevy slot allocation |
| src/workers/continuum-core/src/live/avatar/registry.rs | New backend registry for avatar render backends |
| src/workers/continuum-core/src/live/avatar/publishers/mod.rs | Publisher module wiring (macOS-gated) |
| src/workers/continuum-core/src/live/avatar/mod.rs | Avatar module exports + publisher exports |
| src/workers/continuum-core/src/live/avatar/hash.rs | Deterministic hashing utilities |
| src/workers/continuum-core/src/live/avatar/gender.rs | Voice/identity-based gender selection helpers |
| src/workers/continuum-core/src/live/avatar/frame_analysis.rs | Update test imports to new live/avatar paths |
| src/workers/continuum-core/src/live/avatar/frame.rs | Add HD/FullHD tiers + updated thresholds/tests |
| src/workers/continuum-core/src/live/avatar/backends/procedural.rs | Port procedural backend imports to live/avatar |
| src/workers/continuum-core/src/live/avatar/backends/mod.rs | Backends module exports |
| src/workers/continuum-core/src/live/avatar/backends/live2d.rs | Port Live2D backend imports to live/avatar |
| src/workers/continuum-core/src/live/avatar/backends/bevy_3d.rs | Port Bevy backend imports + MAX_AVATAR_SLOTS references |
| src/workers/continuum-core/src/live/avatar/backend.rs | RenderBackend trait + ModelFormat inference + TS exports |
| src/workers/continuum-core/src/live/audio/vad/webrtc.rs | Add earshot-based WebRTC VAD implementation |
| src/workers/continuum-core/src/live/audio/vad/wav_loader.rs | Add WAV loader helpers for test audio |
| src/workers/continuum-core/src/live/audio/vad/silero_raw.rs | Add raw ONNX Silero VAD implementation |
| src/workers/continuum-core/src/live/audio/vad/rms_threshold.rs | Add RMS-threshold VAD implementation |
| src/workers/continuum-core/src/live/audio/vad/production.rs | Add two-stage ProductionVAD + sentence buffering |
| src/workers/continuum-core/src/live/audio/vad/mod.rs | VAD module API + factory + exports |
| src/workers/continuum-core/src/live/audio/vad/README.md | Document VAD architecture/usage/testing |
| src/workers/continuum-core/src/live/audio/tts_service.rs | Port TTS service to live::audio::tts |
| src/workers/continuum-core/src/live/audio/tts/silence.rs | Add SilenceTTS adapter |
| src/workers/continuum-core/src/live/audio/tts/piper.rs | Add PiperTTS adapter implementation |
| src/workers/continuum-core/src/live/audio/tts/phonemizer.rs | Add espeak-ng phonemizer support |
| src/workers/continuum-core/src/live/audio/tts/audio_utils.rs | Shared resample/normalize utilities for TTS adapters |
| src/workers/continuum-core/src/live/audio/stt_service.rs | Port STT service to live::audio::stt |
| src/workers/continuum-core/src/live/audio/stt/stub.rs | Add Stub STT adapter |
| src/workers/continuum-core/src/live/audio/stt/mod.rs | New STT registry + adapters + API |
| src/workers/continuum-core/src/live/audio/mod.rs | Declare live audio submodules |
| src/workers/continuum-core/src/live/audio/mixer.rs | Port mixer to live handle + new ProductionVAD import |
| src/workers/continuum-core/src/live/audio/buffer.rs | Port buffer Handle import to live/handle |
| src/workers/continuum-core/src/lib.rs | Export live instead of voice and re-export orchestrator |
| src/workers/continuum-core/src/ipc/mod.rs | Wire IPC server state to modules::live + live services |
| src/workers/continuum-core/src/ffi/mod.rs | Port FFI imports to crate::live |
| src/workers/Cargo.toml | Add release codegen-units=1 mitigation comment/config |
| src/widgets/live/public/live-widget.styles.ts | Move captions overlay into .content-area layout |
| src/widgets/live/public/live-widget.scss | Add .content-area + update captions positioning |
| src/widgets/live/public/live-participant-tile.scss | Add hover pin overlay + pinned state styling |
| src/widgets/live/LiveWidget.ts | Persist CC state + pin UX events + connecting gate on state load |
| src/widgets/live/LiveParticipantTile.ts | Add pin button + pinned class + new event dispatch |
| src/widgets/live/LiveControls.ts | Render CC icon as “CC” text inside SVG |
| src/widgets/live/LiveCaptions.ts | Keep captions node mounted; toggle visibility via style |
| src/widgets/chat/chat-widget/chat-widget.css | Style call button in chat header |
| src/system/user/shared/BaseUser.ts | Add personalDbHandle for persona DB isolation |
| src/system/user/server/modules/cognitive/memory/PersonaMemory.ts | Switch to live DB handle getter + require-handle guard |
| src/system/user/server/modules/cognitive/memory/Hippocampus.ts | Set parent personalDbHandle; remove STM-only fallback |
| src/system/user/server/modules/cognition/CognitionLogger.ts | Clean imports/comments around per-persona DB handles |
| src/system/user/server/modules/being/LimbicSystem.ts | Replace db propagation with ensureDbReady() |
| src/system/user/server/PersonaUser.ts | Enforce DB-ready init order before autonomous servicing |
| src/system/data/entities/UserStateEntity.ts | Add persisted captionsEnabled flag |
| src/shared/version.ts | Version bump |
| src/shared/generated/voice/ResolutionTier.ts | Regenerate wire type to include hd/full_hd |
| src/scripts/test-tts-audio.sh | Add TTS capture/playback test script |
| src/scripts/test-persona-speak.sh | Add end-to-end persona speak test script |
| src/scripts/system-stop.sh | Add “nuclear stop” process cleanup script |
| src/scripts/parallel-start.sh | Parallel TS/Rust build + hot restart logic |
| src/scripts/install-livekit.sh | Install helper for LiveKit server |
| src/scripts/download-avatar-models.sh | Download CC0 avatar models |
| src/package.json | Version bump |
| src/package-lock.json | Lockfile version bump |
| src/output.md | Add chat export artifact |
| README.md | Add screenshots/media to README |
Files not reviewed (1)
- src/package-lock.json: Language not supported
Comments suppressed due to low confidence (2)
src/workers/continuum-core/src/live/video/bevy_renderer.rs:543
FRAME_COUNTERis sized with a hard-coded[AtomicU32; 16]and initializer[INIT; 16]. This will become incorrect ifMAX_AVATAR_SLOTSever changes. Tie the array length toMAX_AVATAR_SLOTS as usize(or use a Vec/HashMap keyed by slot) to avoid subtle out-of-bounds bugs.
src/workers/continuum-core/src/live/video/bevy_renderer.rs:60- The AVATAR_FPS throughput comment still references "16 slots × VGA × 15fps ≈ 29 MB/s", but the default resolution is now 1280×720. This makes the performance note misleading; update the comment to match the current default dimensions (or word it in terms of variables).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| this.requestUpdate(); | ||
| }).catch(err => { | ||
| console.error('LiveWidget: Failed to load user context:', err); | ||
| this._stateLoaded = true; // Unblock UI — use defaults |
There was a problem hiding this comment.
In the loadUserContext() error path, _stateLoaded is set to true but requestUpdate() is never called. If isJoined && !_stateLoaded is currently rendering the connecting UI, this can leave the widget stuck there after a load failure. Call this.requestUpdate() (or otherwise trigger an update) in the catch block as well.
Suggested change
| this._stateLoaded = true; // Unblock UI — use defaults | |
| this._stateLoaded = true; // Unblock UI — use defaults | |
| this.requestUpdate(); |
generateBarrelExport assumed filename == type name, but ts-rs can export a different name (e.g. ResolutionTier.ts exports ResolutionTierWire). Now uses parseExportedTypes() to read the real export from each file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
personalDbHandleon BaseUser, parent reference pattern, init order fix, no fallbacksDetails
Video/GPU (Rust)
FramePublishertrait withGpuBridgePublisher(macOS) andReadbackPublisher(cross-platform) implementationsLive Widget (TypeScript/Lit)
position:absoluteat bottom of.content-areacallState, CC button with text iconDB Isolation Fix
BaseUser.personalDbHandle: DbHandle | null— every citizen gets personal storage() => DbHandle | null— no propagation neededTest plan
🤖 Generated with Claude Code