Skip to content

Latest commit

 

History

History
307 lines (206 loc) · 11.2 KB

README.md

File metadata and controls

307 lines (206 loc) · 11.2 KB

Vision Pro Blender Live Streamer

Overview

The Vision Pro Blender Live Streamer is a two-part open-source project that enables real-time streaming of Blender scenes to Apple Vision Pro for immersive spatial preview. Create and edit in Blender, see results instantly in your Vision Pro.

This project consists of:

  1. Blender Add-on (Python): Handles Bonjour/Zeroconf discovery, TCP connection, and real-time USDZ export
  2. Vision Pro Application (Swift/RealityKit): Advertises via Bonjour, receives USDZ data, and renders in immersive space

Features

  • Automatic Discovery: Finds Vision Pro devices on your local network via Bonjour/Zeroconf
  • One-Click Installation: Auto-installs required dependencies (python-zeroconf) from within Blender
  • Activity-Based Streaming: Streams only when you're actively editing, pauses during idle periods to save resources
  • Real-time USDZ Export: Exports scenes with materials, textures, and geometry optimized for Apple platforms
  • Configurable Frame Rate: Control streaming speed (1-60 FPS)
  • Immersive Mixed Reality: Renders scenes in Vision Pro's mixed immersion mode

Requirements

  • Blender: 5.0.0 or newer
  • Apple Vision Pro: visionOS 1.0 or later
  • Network: Both devices on the same local network (Wi-Fi)
  • Python Library: python-zeroconf (auto-installed by add-on)

Installation

1. Blender Add-on Setup

Quick Install (Recommended)

  1. Download the Blender folder containing:

    • vision_pro_streamer_zeroconf.py (main add-on)
    • Utilities/auto_install_zeroconf.py (optional dependency installer)
    • Utilities/uninstall_zeroconf.py (optional dependency remover)

  2. Install the add-on:

    • Open Blender
    • Go to Edit > Preferences > Add-ons
    • Click "Install..." and select vision_pro_streamer_zeroconf.py
    • Enable the add-on: "Development: Vision Pro Streamer (Zeroconf)"

  3. Auto-install dependencies:

    • The add-on will attempt to auto-install python-zeroconf on first use

    • If auto-install fails, run Utilities/auto_install_zeroconf.py with Blender's Python:

      • Option A: Open Blender's Scripting workspace, open Utilities/auto_install_zeroconf.py, and click "Run Script"

      • Option B: Run from terminal: (Note: Change paths to Blender and Python files where needed)

        # macOS
/Applications/Blender.app/Contents/MacOS/Blender --background --python /path/to/Utilities/auto_install_zeroconf.py

# Windows
"C:\Program Files\Blender Foundation\Blender 5.0\blender.exe" --background --python "C:\path\to\Utilities\auto_install_zeroconf.py"

# Linux
~/blender-5.0.0-linux-x64/blender --background --python /path/to/Utilities/auto_install_zeroconf.py

    • Restart Blender after installation

Manual Install (Fallback)

If auto-installation fails:

  1. Locate Blender's Python:

    • macOS: /Applications/Blender.app/Contents/Resources/5.0/python/bin/python3.11
    • Windows: C:\Program Files\Blender Foundation\Blender 5.0\5.0\python\bin\python.exe
    • Linux: ~/blender-5.0.0-linux-x64/5.0/python/bin/python3

  2. Note: Replace 5.0 with your installed Blender version (e.g., 4.4, 4.5, etc.)

  3. Install via terminal:

    "<path_to_blender_python>" -m pip install --user zeroconf

  4. Verify: Check Blender's System Console (Window > Toggle System Console) for "✓ zeroconf loaded"

  5. Note: The add-on's warning message "Requires a Vision Pro application..." in Preferences is informational only and will always appear, even when zeroconf is properly installed.

2. Vision Pro Application Setup

  1. Open the Xcode project:

    • Open VisionPro Blender Streamer.xcodeproj in Xcode

  2. Configure Info.plist (Already configured, but verify):

    • NSBonjourServices contains: _visionpro_blender._tcp. and _visionpro_blender._udp.
    • NSLocalNetworkUsageDescription is set for local network permissions

  3. Build and Deploy:

    • Select your Vision Pro device or Simulator
    • Build and run (⌘R)
    • Grant local network permissions when prompted

  4. Note: The app advertises as "MyVisionPro" on port 8080

Usage

Quick Start

  1. Launch Vision Pro App

    • Open the app on your Vision Pro
    • Tap "Toggle Immersive Space" to enter mixed reality mode
    • App automatically advertises as "MyVisionPro" and listens on port 8080

  2. Open Blender

    • Launch Blender with the add-on enabled
    • Press N in the 3D Viewport to open the sidebar
    • Navigate to the "Vision Pro" tab

  3. Connect to Vision Pro

    • Click "Start Discovery"
    • Wait for "MyVisionPro" to appear in the dropdown (usually 2-5 seconds)
    • Select your device from the list
    • Click "Connect"
    • Status should show: "Connected to MyVisionPro"

  4. Start Streaming

    • Click "Start Streaming"
    • Your scene will appear in Vision Pro's immersive space
    • Make changes in Blender to see real-time updates

Streaming Options

Stream FPS: Controls update frequency (1-60 FPS, default: 30)

  • Higher FPS = smoother updates, more CPU usage
  • Lower FPS = less resource intensive

Stream Only When Active: (Recommended)

  • ☑ Enabled: Streams only when you're editing (moving objects, modifying meshes, etc.)
  • ☐ Disabled: Streams continuously at set FPS
  • Inactivity Threshold: Seconds of idle time before pausing (default: 2.0s)

Workflow Tips

  • Debugging: Open System Console (Window > Toggle System Console on Windows, or launch from Terminal on macOS/Linux)
  • Performance: Enable "Stream Only When Active" to reduce CPU/network load
  • Scene Position: Models appear 1m up, 2m forward from your position in Vision Pro
  • Stopping: Click "Stop Streaming" to pause, "Disconnect" to close connection

Troubleshooting

Dependency Issues

"zeroconf not found" warning:

  1. Run Utilities/auto_install_zeroconf.py with Blender's Python:

    • Open Blender's Scripting workspace, open the file, and click "Run Script"
    • Or run from terminal: /Applications/Blender.app/Contents/MacOS/Blender --background --python /path/to/Utilities/auto_install_zeroconf.py

  2. Check System Console for installation errors

  3. If auto-install fails, use manual installation method (see Installation)

  4. Restart Blender after installation

  5. To uninstall, run Utilities/uninstall_zeroconf.py the same way:

    /Applications/Blender.app/Contents/MacOS/Blender --background --python /path/to/Utilities/uninstall_zeroconf.py

Connection Issues

"Connection failed" / "BrokenPipeError":

  • ✓ Vision Pro app is running and in immersive space
  • ✓ Both devices on same Wi-Fi network (not cellular/hotspot)
  • ✓ Vision Pro granted "Local Network" permission (Settings > Privacy)
  • ✓ Port 8080 is not blocked by firewall
  • ✓ Service type matches: _visionpro_blender._tcp.

No devices found in discovery:

  • Wait 10-15 seconds after clicking "Start Discovery"
  • Restart Vision Pro app
  • Click "Stop Discovery" then "Start Discovery" again
  • Check System Console for Bonjour errors
  • Verify Info.plist has NSBonjourServices with both _visionpro_blender._tcp. and _visionpro_blender._udp.

Streaming Issues

"USDZ Export Error: keyword unrecognized":

  • Your Blender version may have different USD export parameters
  • Edit vision_pro_streamer_zeroconf.py, find bpy.ops.wm.usd_export()
  • Remove problematic parameters (e.g., usdz_downscale_size)
  • Consult Blender Python API docs for your version

Scene not updating in Vision Pro:

  • Check "Realtime Status" message in Blender panel
  • Verify "Stream Only When Active" isn't pausing due to inactivity
  • Ensure you're making changes that trigger depsgraph updates
  • Try disabling "Stream Only When Active" for continuous streaming
  • Large or complex scenes may take longer to export and transfer—check System Console for timing

Performance issues:

  • Lower Stream FPS (try 15-20 FPS)
  • Enable "Stream Only When Active"
  • Reduce scene complexity: lower poly counts, simplify materials, or optimize textures

UI Issues

Device list empty/UI sections disappear:

  • Disable and re-enable add-on in Preferences
  • Check System Console for Python errors
  • Ensure enum_items_cache is updating (debug output in console)
  • Restart Blender if UI becomes unresponsive

Technical Details

Architecture

Blender Add-on:

  • Zeroconf service browser for device discovery
  • TCP client connecting to Vision Pro on port 8080
  • Depsgraph handler for activity detection
  • Threaded USDZ export and streaming
  • Temporary file management for each export cycle

Vision Pro App:

  • NetService (Bonjour) advertiser
  • NWListener (Network framework) TCP server
  • AsyncStream for entity updates to RealityKit
  • Temporary USDZ file processing
  • Mixed immersion mode rendering

Data Flow

  1. Vision Pro advertises _visionpro_blender._tcp. and _visionpro_blender._udp. on port 8080
  2. Blender discovers service via Zeroconf
  3. Blender establishes TCP connection
  4. On scene change (or timer), Blender exports USDZ to temp file
  5. USDZ data sent with 4-byte length header (big-endian)
  6. Vision Pro receives, writes to temp file, loads as Entity
  7. Entity replaces previous scene in immersive space

Customization

Change service name: Edit VisionProServiceAdvertiser.swift, line 16:

netService = NetService(domain: "local.", type: serviceType, name: "YourCustomName", port: servicePort)

Change port: Update both files:

  • VisionProServiceAdvertiser.swift: servicePort = 8080
  • BlenderSceneReceiver.swift: init(port: 8080)

Adjust scene position: Edit ImmersiveView.swift, line 32:

dynamicContentAnchor.transform.translation = SIMD3<Float>(x: 0.0, y: 1.0, z: -2.0)

Licensing

This project is licensed under the BSD 3-Clause License.

Copyright (c) 2025-2026, Southwest Airlines All rights reserved.

Note on Blender Add-on Licensing: Blender is licensed under the GNU GPL. Add-ons using Blender's Python API may be considered derivative works under some GPL interpretations. For commercial distribution, consult legal counsel or consider GPLv3 licensing.

Known Limitations

  • Network Only: Requires both devices on same local network (no internet/cloud streaming)
  • USDZ Format: Limited to features supported by USD/USDZ (some Blender features may not export)
  • Single Connection: Vision Pro app accepts one Blender connection at a time
  • No Animation Playback: Exports static scenes only (animation export disabled for performance)
  • Temporary Files: Creates temp files for each export (cleaned up automatically)
  • Scene Complexity: High-poly models or complex materials may take longer to export and stream—start with simpler scenes for best experience

Acknowledgements

  • Uses python-zeroconf library (LGPL-2.1-or-later)
  • Built on Blender Python API and Apple VisionOS/RealityKit frameworks
  • Created by Justin Leger (2025)

Contributing

Contributions welcome! Please:

  • Open issues for bugs or feature requests
  • Submit pull requests with clear descriptions
  • Follow existing code style and structure
  • Test on both Blender and Vision Pro before submitting