Skip to content

Commit fad7045

Browse files
De-PantherDe-Panther
authored
Added docs for WebXR Mixed Reality Capture
1 parent 71e2d35 commit fad7045

File tree

1 file changed

+94
-0
lines changed

1 file changed

+94
-0
lines changed

‎Documentation/WebXR-Mixed-Reality-Capture.md

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# WebXR Mixed Reality Capture
2+
3+
Using Mixed Reality Capture in WebXR requires different parts to work together. In this page we will go over the different parts and settings to make it work in a project.
4+
5+
## General info
6+
7+
Mixed Reality Capture is used to merge between the real world and the virtual world, by positioning a real camera feed inside the virtual environment and displaying the outcome in a spectator display.
8+
In this WebXR solution, we made sure that WebXR Export can support Spectator mode and had the code of the Mixed Reality Capture feature in the WebXR Interactions package.
9+
A configured prefab can be found in the Sample scene of WebXR Interactions named as `SpectatorCameraHolder.prefab`.
10+
11+
## Expected hierarchy
12+
13+
The current configuration is expected for Mixed Reality Capture to work properly
14+
15+
```
16+
- SpectatorCameraHolder
17+
- SpectatorCamera (Components: Camera, SpectatorCamera)
18+
- StackCameras
19+
- SpectatorBackgroundCamera (Components: Camera)
20+
- SpectatorWebcamLightingCamera (Components: Camera)
21+
- SpectatorForegroundCamera (Components: Camera)
22+
- Background (Components: MeshFilter with Quad, MeshRenderer with UnlitTransparent Material. Layer: Spectator)
23+
- Foreground (Components: MeshFilter with Quad, MeshRenderer with UnlitTransparent Material. Layer: Spectator)
24+
- MixedRealityCaptureController (Components: MixedRealityCaptureController)
25+
- CameraPoint (Used as Visual reference for moving point)
26+
- TopPoint (Used as Visual reference for moving point)
27+
- BottomPoint (Used as Visual reference for moving point)
28+
- WebcamHolder
29+
- WebcamQuad (Components: PlayWebcam, MeshFilter with Quad, MeshRenderer with ChromaKeyUnlit Material. Layer: Webcam)
30+
- WebcamLightingQuad (Components: MeshFilter with Quad, MeshRenderer with White Legacy Diffuse Material. Layer: WebcamLighting)
31+
- CameraHint (Used as Visual reference for point on webcam)
32+
- TopHint (Used as Visual reference for point on webcam)
33+
- BottomHint (Used as Visual reference for point on webcam)
34+
```
35+
36+
## Layers
37+
38+
The `SpectatorCameraHolder.prefab` sample prefab uses pre-defined layers:
39+
40+
- 16 Webcam
41+
- 17 WebcamLighting
42+
- 18 Spectator
43+
44+
## Configuring Mixed Reality Capture in a project
45+
46+
Some general steps to follow when implementing WebXR Mixed Reality Capture in a project:
47+
48+
- Import the `SpectatorCameraHolder.prefab` from the sample. It's better to duplicate it and using the duplicated asset.
49+
- Add the prefab to the scene with the WebXR Camera Rig (or `WebXRCameraSet`).
50+
- If the camera rig is expected to be static, the `SpectatorCameraHolder` can be at the root of the scene, else it should be a child of the camera rig.
51+
- Make sure `MixedRealityCaptureController` is configured correctly.
52+
- `webcamLayer` point to the Webcam layer.
53+
- `mixedRealityOnLayers` are the layers that the Spectator camera would use during the capture mode, should contain the Webcam and Spectator layers.
54+
- `camerasBase` reference to the Cameras Transform in the camera rig hierarchy.
55+
- `cameraFollower` reference the `CameraFollower` Transform in the camera rig hierarchy.
56+
- `xrCameras` reference all the XR cameras Camera components in the camera rig hierarchy.
57+
- `spectatorCamera` reference the Spectator camera Camera in the `SpectatorCameraHolder` hierarchy.
58+
- `stackCameras` reference the `StackCameras` GameObject in the `SpectatorCameraHolder` hierarchy.
59+
- `spectatorBackgroundCamera` reference the `SpectatorBackgroundCamera` Camera in the `SpectatorCameraHolder` hierarchy.
60+
- `spectatorForegroundCamera` reference the `SpectatorForegroundCamera` Camera in the `SpectatorCameraHolder` hierarchy.
61+
- `spectatorWebcamLightingCamera` reference the `SpectatorWebcamLightingCamera` Camera in the `SpectatorCameraHolder` hierarchy.
62+
- `spectatorCameraTransform` reference the Spectator camera Transform in the `SpectatorCameraHolder` hierarchy.
63+
- `spectatorCameraParent` reference the `SpectatorCameraHolder` Transform.
64+
- `backgroundPlaneRenderer` reference the `Background` Renderer in the `SpectatorCameraHolder` hierarchy.
65+
- `foregroundPlaneRenderer` reference the `Foreground` Renderer in the `SpectatorCameraHolder` hierarchy.
66+
- `backgroundPlaneTransform` reference the `Background` Transform in the `SpectatorCameraHolder` hierarchy.
67+
- `foregroundPlaneTransform` reference the `Foreground` Transform in the `SpectatorCameraHolder` hierarchy.
68+
- `defaultPlaneMaterial` default Material used for the background and foreground planes, Unlit Transparent material.
69+
- `webcamParent` reference the `WebcamHolder` Transform in the `SpectatorCameraHolder` hierarchy.
70+
- `webcam` reference the `WebcamQuad` PlayWebcam in the `SpectatorCameraHolder` hierarchy.
71+
- `calibrationPointCamera` reference the `CameraPoint` Transform in the `SpectatorCameraHolder` hierarchy.
72+
- `calibrationPointTop` reference the `TopPoint` Transform in the `SpectatorCameraHolder` hierarchy.
73+
- `calibrationPointBottom` reference the `BottomPoint` Transform in the `SpectatorCameraHolder` hierarchy.
74+
- `calibrationHintCamera` reference the `CameraHint` GameObject in the `SpectatorCameraHolder` hierarchy.
75+
- `calibrationHintTop` reference the `TopHint` GameObject in the `SpectatorCameraHolder` hierarchy.
76+
- `calibrationHintBottom` reference the `BottomHint` GameObject in the `SpectatorCameraHolder` hierarchy.
77+
- `leftController` reference the left WebXRController in the camera rig hierarchy.
78+
- `rightController` reference the right WebXRController in the camera rig hierarchy.
79+
- `webcamFramesDelaySize` set the default number of delayed frames to store - Used for cases of Webcam image slower than VR movement.
80+
- Make sure that the Main Camera and XR Cameras hide or show the correct layers, the Spectator, Webcam and WebcamLighting should be hidden.
81+
- Make sure that the Spectator camera shows the Spectator and Webcam Layers and other layers that the project should display.
82+
- Make sure that the Spectator camera hides the WebcamLighting layer.
83+
- Make sure that the background and foreground cameras hide the Spectator, Webcam and WebcamLighting layers.
84+
- Make sure that the background and foreground cameras show all other layers that the project should display.
85+
- Make sure that the SpectatorWebcamLightingCamera only shows the WebcamLighting layer.
86+
87+
That should handle all the configurations needed for Mixed Reality Captuer to work properly.
88+
89+
## Enable Mixed Reality Captuer on runtime
90+
91+
Once the project runs, for the feature to be enabled, both the `SpectatorCamera` and `MixedRealityCaptureController` should be set to be enable in XR.
92+
Both components contains `EnableInXR(bool)` method to enable their functionality when user is switching to XR mode.
93+
94+
There are more customization and settings that can be handle by scripts like Chroma Key min and max range, check the `MixedRealityCaptureController` and `PlayWebcam` code for those options.

0 commit comments

Comments
 (0)