Warning
This documentation is outdated and will be updated in the future. The code structure has changed significantly since this was written, so please refer to the code itself for the most up-to-date information.
The code of Spatialstrates is modular and organized into the structure below. The functionality is structured into five main folders which each include multiple WPM (Webstrates Package Manager) packages.
Each package defines their own dependencies, and the WPM GUI in Cauldron can be used to define which packages should be loaded. For instance, if the WebRTC streaming features are not required, one can simply tick off the StreamingBundle package in the WPM GUI to avoid loading it.
This folder contains four WPM bundles, which each include multiple packages. This makes importing certain types of functionality easier, as less individual packages have to be imported.
The minimum recommended set of packages to include when starting a new Spatialstrates project.
Some useful additions to Spatialstrates.
WebRTC streaming functionality for audio, video, and screen sharing.
- Stream Manager
- Screen Stream Manager
- Video Stream Manager
- Audio Stream Manager
- Screen Stream
- Video Stream
- Audio Stream
The base Movable implementation and basic components.
The main components of Spatialstrates that bootstrap the system.
Includes the import map used in import statements. It uses the esm.sh CDN for serving ES modules. As the import map script needs to be present in the DOM on load, this package needs to be embedded into a webstrate in order to work properly.
For development we use the ?dev version of modules in the import map. We also use the * operator in front of modules, which prevents the import of dependencies. This is necessary as some packages otherwise import different versions of React or R3F, which causes all kinds of issues. For some packages, like Vega or D3, we omit the * operator as those are largely self-contained and so far did not cause conflicts with other packages.
If additional packages are needed, they can be added to the import map here.
The main package that bootstraps Spatialstrates. The React Reloader fragment creates a React root and initializes it with the App component. It also listens for changes to the App component and the Varv model and reloads Spatialstrates when either of them changes.
The App component creates the R3F scene and XR session. It handles what basic elements are visible in the scene, e.g., lighting and the grey background. Apart from that, the package includes basic styling and the Camera, a custom camera that can be controlled using the mouse and keyboard.
It also adds the Dynamic Components mechanism, which enables "hot-(re)loading" of new components while Spatialstrates is running. See the Dynamic Components API for how to add dynamic components to Spatialstrates.
These packages add some basic functionality like menus to toggle XR to Spatialstrates.
The 2D menu at the bottom of a spatialstrate. It includes a simple API to add menu groups and entries. See the Menu API on how to use it.
The menu that is displayed on top of a controller when in immersive XR. Like the normal menu, it includes a simple API to add menu entries. See the Controller Menu API on how to use it.
The basic menu entries to move the camera, toggle AR and VR, and to send the link of a spatialstrate to one's Quest headset.
In this package also the parameters and features for the WebXR session are defined.
A helper utility that renders 3D icons that can be themed and used in the scene. Color themes of icons need to be supported by the 3D models for this to work.
A utility that renders text in the 3D scene. It is a wrapper for the R3F Drei "Text" component that sets the font to a custom font. It can also be helpful when debugging to globally disable text rendering in the scene to investigate its performance impact. (Text rendering is quite expensive.)
Various useful add-ons for Spatialstrates.
A small package that adds the header logo in the top left corner of Spatialstrates and indicates the version number.
A utility that renders a tringular object in immersive XR that allows to calibrate the scene. This is required, as WebXR does not yet support persisted anchors and the XR scene always starts with the origin at the center of the headset where it was started from.
The calibration point can be moved using the motion controller and should be aligned with a salient point in the physical world, e.g., a corner of a table. It should also be placed close to the center of the room or 3D scene, as small errors in the alignment rotation can lead to large offsets in the scene the further away the calibration point is.
A small user manager utility that allows to add and select users. It is used by the avatar functionality to display the name of a user above their avatar.
The user manager uses a hard-coded default user that is always selected when starting Spatialstrates. It can be changed by opening the user manager dialog with the button in the top right corner.
The user is not stored between sessions and has to be selected again when starting Spatialstrates.
The avatar functionality that displays an avatar for each user in the scene. Avatars are generated for the camera of each user and the controllers of users in immersive XR. Different models are used for the desktop users, phone users, and HMD users. Avatars can be toggled on and off using the menu. Toggling them on or off is a global action that affects all users.
Currently, controller and HMD models always show the Meta Quest 3 models. For broader compatibility, more models and a better detection of the user agent should be added.
Note
Using avatars impacts the performance of Spatialstrates on the Quest 3 and Quest Pro significantly.
A small set of helper functions to access the transcribe audio using Whisper and to create completion calls towards the OpenAI API.
These packages provide the functionality to stream audio and video in Spatialstrates.
This folder includes the dynamic components for the streaming functionality. Screen streams are rendered in the scene as a Movable object, video streams are placed on top of user's avatars, and audio streams are invisible but transmit the microphone input to other clients. These components also add menu entries to start and stop each of the stream types.
This folder includes a basic StreamManager class that is initialized for audio, video, and screen streams. These managers manage the signal streams created using the Webstrates signal streaming API. They also provide helper functions on the window object to start and stop the streams.
Movables are a class of components that are placed in the 3D scene and that have their position and rotation persisted and synchronized using Varv. They can be moved both using the mouse and keyboard, and using motion controllers or hand tracking in immersive XR.
The mechanism enabling Movables is quite complex and should not be changed unless necessary. It uses a mix of R3F components and Varv actions to facilitate shared 3D objects in the scene. In the future, the Varv part probably will be removed to improve performance.
This component is necessary, as the state of the 3D scene is otherwise not synchronized among clients. Hence, all other 3D objects placed in the scene are by default transient and do not share state with other clients.
This package includes the basic functionality of Movables.
A simple virtual sticky note component that can be moved around in the scene. It can be modified using the microphone icon to transcribe audio.
This component is a good example of how to create components that use the Movable component, due to its simple structure.
A simple image component. Images can, so far, only be added using the "Upload Image" menu entry. The uploaded image is then added as a Webstrate asset and placed in the scene.
The trashcan is another simple component that can be used to delete other components from the scene. It is a good example of how to create components that interact with other movables in the scene.
This package includes a small screenshot utility that adds a button to the menu, which takes a screenshot of the scene, uploads it as an Webstrate asset, and places it in the 3D scene as an image component.