Merge pull request #110182 from MicrosoftDocs/release-preview-arr

Release preview arr

Author: huypub

articles/index.yml

@@ -1188,6 +1188,13 @@ productDirectory:
         - mixed-reality
       url: spatial-anchors/index.yml
     # Card
+    - title: Remote Rendering (Preview)
+      summary: Render high-quality, interactive 3D content, and stream it to your devices in real time
+      imageSrc: ./media/index/remote-rendering.svg
+      azureCategories:
+        - mixed-reality
+      url: remote-rendering/index.yml
+    # Card
     - title: Visual Studio App Center
       summary: Continuously build, test, release, and monitor your mobile and desktop apps
       imageSrc: https://docs.microsoft.com/media/logos/logo_vs-mobile-center.svg

articles/media/index/remote-rendering.svg

@@ -0,0 +1,48 @@
+---
+title: Components
+description: Definition of components in the scope of Azure Remote Rendering
+author: florianborn71
+ms.author: flborn
+ms.date: 02/04/2020
+ms.topic: conceptual
+---
+
+# Components
+
+Azure Remote Rendering uses the [Entity Component System](https://en.wikipedia.org/wiki/Entity_component_system) pattern. While [entities](entities.md) represent the position and the hierarchical composition of objects, components are responsible for implementing behavior.
+
+The most frequently used types of components are [mesh components](meshes.md), which add meshes into the rendering pipeline. Similarly, [light components](../overview/features/lights.md) are used to add lighting and [cut plane components](../overview/features/cut-planes.md) are used to cut open meshes.
+
+All these components use the transform (position, rotation, scale) of the entity they are attached to, as their reference point.
+
+## Working with components
+
+You can easily add, remove, and manipulate components programmatically:
+
+```cs
+// create a point light component
+AzureSession session = GetCurrentlyConnectedSession();
+PointLightComponent lightComponent = session.Actions.CreateComponent(ObjectType.PointLightComponent, ownerEntity) as PointLightComponent;
+
+lightComponent.Color = new Color4Ub(255, 150, 20, 255);
+lightComponent.Intensity = 11;
+
+// ...
+
+// destroy the component
+lightComponent.Destroy();
+lightComponent = null;
+```
+
+A component is attached to an entity at creation time. It cannot be moved to another entity afterwards. Components are explicitly deleted with `Component.Destroy()` or automatically when the component's owner entity is destroyed.
+
+Only one instance of each component type may be added to an entity at a time.
+
+## Unity specific
+
+The Unity integration has additional extension functions for interacting with components. See [Unity game objects and components](../how-tos/unity/objects-components.md).
+
+## Next steps
+
+* [Object bounds](object-bounds.md)
+* [Meshes](meshes.md)

articles/remote-rendering/concepts/components.md

@@ -0,0 +1,82 @@
+---
+title: Entities
+description: Definition of entities in the scope of the Azure Remote Rendering API
+author: florianborn71
+ms.author: flborn
+ms.date: 02/03/2020
+ms.topic: conceptual
+---
+
+# Entities
+
+An *Entity* represents a movable object in space and is the fundamental building block of remotely rendered content.
+
+## Entity properties
+
+Entities have a transform defined by a position, rotation, and scale. By themselves entities do not have any observable functionality. Instead, behavior is added through components, which are attached to entities. For instance, attaching a [CutPlaneComponent](../overview/features/cut-planes.md)  will create a cut plane at the position of the entity.
+
+The most important aspect of the entity itself is the hierarchy and the resulting hierarchical transform. For example, when multiple entities are attached as children to a shared parent entity, all of these entities can be moved, rotated, and scaled in unison by changing the transform of the parent entity.
+
+An entity is uniquely owned by its parent, meaning that when the parent is destroyed with `Entity.Destroy()`, so are its children and all connected [components](components.md). Thus, removing a model from the scene is accomplished by calling `Destroy` on the root node of a model, returned by `AzureSession.Actions.LoadModelAsync()` or its SAS variant `AzureSession.Actions.LoadModelFromSASAsync()`.
+
+Entities are created when the server loads content or when the user wants to add an object to the scene. For example, if a user wants to add a cut plane to visualize the interior of a mesh, the user can create an entity where the plane should exist and then add the cut plane component to it.
+
+## Query functions
+
+There are two types of query functions on entities: synchronous and asynchronous calls. Synchronous queries can only be used for data that is present on the client and does not involve much computation. Examples are querying for components, relative object transforms, or parent/child relationships. Asynchronous queries are used for data that only resides on the server or involves extra computation that would be too expensive to run on the client. Examples are spatial bounds queries or meta data queries.
+
+### Querying components
+
+To find a component of a specific type, use `FindComponentOfType`:
+
+```cs
+CutPlaneComponent cutplane = (CutPlaneComponent)entity.FindComponentOfType(ObjectType.CutPlaneComponent);
+
+// or alternatively:
+CutPlaneComponent cutplane = entity.FindComponentOfType<CutPlaneComponent>();
+```
+
+### Querying transforms
+
+Transform queries are synchronous calls on the object. It is important to note that transforms queried through the API are local space transforms, relative to the object's parent. Exceptions are root objects, for which local space and world space are identical.
+
+> [!NOTE]
+> There is no dedicated API to query the world space transform of arbitrary objects.
+
+```cs
+// local space transform of the entity
+Double3 translation = entity.Position;
+Quaternion rotation = entity.Rotation;
+```
+
+### Querying spatial bounds
+
+Bounds queries are asynchronous calls that operate on a full object hierarchy, using one entity as a root. See the dedicated chapter about [object bounds](object-bounds.md).
+
+### Querying metadata
+
+Metadata is additional data stored on objects, that is ignored by the server. Object metadata is essentially a set of (name, value) pairs, where _value_ can be of numeric, boolean or string type. Metadata can be exported with the model.
+
+Metadata queries are asynchronous calls on a specific entity. The query only returns the metadata of a single entity, not the merged information of a sub graph.
+
+```cs
+MetadataQueryAsync metaDataQuery = entity.QueryMetaDataAsync();
+metaDataQuery.Completed += (MetadataQueryAsync query) =>
+{
+    if (query.IsRanToCompletion)
+    {
+        ObjectMetaData metaData = query.Result;
+        ObjectMetaDataEntry entry = metaData.GetMetadataByName("MyInt64Value");
+        System.Int64 intValue = entry.AsInt64;
+
+        // ...
+    }
+};
+```
+
+The query will succeed even if the object does not hold any metadata.
+
+## Next steps
+
+* [Components](components.md)
+* [Object bounds](object-bounds.md)

articles/remote-rendering/concepts/entities.md

@@ -0,0 +1,155 @@
+---
+title: Graphics binding
+description: Setup of graphics bindings and use cases
+author: florianborn71
+manager: jlyons
+services: azure-remote-rendering
+titleSuffix: Azure Remote Rendering
+ms.author: flborn
+ms.date: 12/11/2019
+ms.topic: conceptual
+ms.service: azure-remote-rendering
+---
+# Graphics binding
+
+To be able to use Azure Remote Rendering in a custom application, it needs to be integrated into the application's rendering pipeline. This integration is the responsibility of the graphics binding.
+
+Once set up, the graphics binding gives access to various functions that affect the rendered image. These functions can be separated into two categories: general functions that are always available and specific functions that are only relevant for the selected `Microsoft.Azure.RemoteRendering.GraphicsApiType`.
+
+## Graphics binding in Unity
+
+In Unity, the entire binding is handled by the `RemoteUnityClientInit` struct passed into `RemoteManagerUnity.InitializeManager`. To set the graphics mode, the `GraphicsApiType` field has to be set to the chosen binding. The field will be automatically populated depending on whether an XRDevice is present. The behavior can be manually overridden with the following behaviors:
+
+* **HoloLens 2**: the [Windows Mixed Reality](#windows-mixed-reality) graphics binding is always used.
+* **Flat UWP desktop app**: [Simulation](#simulation) is always used. To use this mode, make sure to follow the steps in [Tutorial: Setting up a Unity project from scratch](../tutorials/unity/project-setup.md).
+* **Unity editor**: [Simulation](#simulation) is always used unless a WMR VR headset is connected in which case ARR will be disabled to allow to debug the non-ARR related parts of the application. See also [holographic remoting](../how-tos/unity/holographic-remoting.md).
+
+The only other relevant part for Unity is accessing the [basic binding](#access), all the other sections below can be skipped.
+
+## Graphics binding setup in custom applications
+
+To select a graphics binding, take the following two steps: First, the graphics binding has to be statically initialized when the program is initialized:
+
+``` cs
+RemoteRenderingInitialization managerInit = new RemoteRenderingInitialization;
+managerInit.graphicsApi = GraphicsApiType.WmrD3D11;
+managerInit.connectionType = ConnectionType.General;
+managerInit.right = ///...
+RemoteManagerStatic.StartupRemoteRendering(managerInit);
+```
+
+The call above is necessary to initialize Azure Remote Rendering into the holographic APIs. This function must be called before any holographic API is called and before any other Remote Rendering APIs are accessed. Similarly, the corresponding de-init function `RemoteManagerStatic.ShutdownRemoteRendering();` should be called after no holographic APIs are being called anymore.
+
+## <span id="access">Accessing graphics binding
+
+Once a client is set up, the basic graphics binding can be accessed with the `AzureSession.GraphicsBinding` getter. As an example, the last frame statistics can be retrieved like this:
+
+``` cs
+AzureSession currentSesson = ...;
+if (currentSesson.GraphicsBinding)
+{
+    FrameStatistics frameStatistics;
+    if (session.GraphicsBinding.GetLastFrameStatistics(out frameStatistics) == Result.Success)
+    {
+        ...
+    }
+}
+```
+
+## Graphic APIs
+
+There are currently two graphics APIs that can be selected, `WmrD3D11` and `SimD3D11`. A third one `Headless` exists but is not yet supported on the client side.
+
+### Windows Mixed Reality
+
+`GraphicsApiType.WmrD3D11` is the default binding to run on HoloLens 2. It will create the `GraphicsBindingWmrD3d11` binding. In this mode Azure Remote Rendering hooks directly into the holographic APIs.
+
+To access the derived graphics bindings, the base `GraphicsBinding` has to be cast.
+There are two things that need to be done to use the WMR binding:
+
+#### Inform Remote Rendering of the used coordinate system
+
+``` cs
+AzureSession currentSesson = ...;
+IntPtr ptr = ...; // native pointer to ISpatialCoordinateSystem
+GraphicsBindingWmrD3d11 wmrBinding = (currentSession.GraphicsBinding as GraphicsBindingWmrD3d11);
+if (binding.UpdateUserCoordinateSystem(ptr) == Result.Success)
+{
+    ...
+}
+```
+
+Where the above `ptr` must be a pointer to a native `ABI::Windows::Perception::Spatial::ISpatialCoordinateSystem` object that defines the world space coordinate system in which coordinates in the API are expressed in.
+
+#### Render remote image
+
+At the start of each frame the remote frame needs to be rendered into the back buffer. This is done by calling `BlitRemoteFrame`, which will fill both color and depth information into the currently bound render target. Thus it is important that this is done after binding the back buffer as a render target.
+
+``` cs
+AzureSession currentSesson = ...;
+GraphicsBindingWmrD3d11 wmrBinding = (currentSession.GraphicsBinding as GraphicsBindingWmrD3d11);
+binding.BlitRemoteFrame();
+```
+
+### Simulation
+
+`GraphicsApiType.SimD3D11` is the simulation binding and if selected it creates the `GraphicsBindingSimD3d11` graphics binding. This interface is used to simulate head movement, for example in a desktop application and renders a monoscopic image.
+The setup is a bit more involved and works as follows:
+
+#### Create proxy render target
+
+Remote and local content needs to be rendered to an offscreen color / depth render target called 'proxy' using
+the proxy camera data provided by the `GraphicsBindingSimD3d11.Update` function. The proxy must match the resolution of the back buffer. Once a session is ready, `GraphicsBindingSimD3d11.InitSimulation` needs to be called before connecting to it:
+
+``` cs
+AzureSession currentSesson = ...;
+IntPtr d3dDevice = ...; // native pointer to ID3D11Device
+IntPtr color = ...; // native pointer to ID3D11Texture2D
+IntPtr depth = ...; // native pointer to ID3D11Texture2D
+float refreshRate = 60.0f; // Monitor refresh rate up to 60hz.
+bool flipBlitRemoteFrameTextureVertically = false;
+bool flipReprojectTextureVertically = false;
+GraphicsBindingSimD3d11 simBinding = (currentSession.GraphicsBinding as GraphicsBindingSimD3d11);
+simBinding.InitSimulation(d3dDevice, depth, color, refreshRate, flipBlitRemoteFrameTextureVertically, flipReprojectTextureVertically);
+```
+
+The init function needs to be provided with pointers to the native d3d-device as well as to the color and depth texture of the proxy render target. Once initialized, `AzureSession.ConnectToRuntime` and `DisconnectFromRuntime` can be called multiple times but when switching to a different session, `GraphicsBindingSimD3d11.DeinitSimulation` needs to be called first on the old session before `GraphicsBindingSimD3d11.InitSimulation` can be called on another session.
+
+#### Render loop update
+
+The render loop update consists of multiple steps:
+
+1. Each frame, before any rendering takes place, `GraphicsBindingSimD3d11.Update` is called with the current camera transform that is sent over to the server to be rendered. At the same time the returned proxy transform should be applied to the proxy camera to render into the proxy render target.
+If the returned proxy update `SimulationUpdate.frameId` is null, there is no remote data yet. In this case, instead of rendering into the proxy render target, any local content should be rendered to the back buffer directly using the current camera data and the next two steps are skipped.
+1. The application should now bind the proxy render target and call `GraphicsBindingSimD3d11.BlitRemoteFrameToProxy`. This will fill the remote color and depth information into the proxy render target. Any local content can now be rendered onto the proxy using the proxy camera transform.
+1. Next, the back buffer needs to be bound as a render target and `GraphicsBindingSimD3d11.ReprojectProxy` called at which point the back buffer can be presented.
+
+``` cs
+AzureSession currentSesson = ...;
+GraphicsBindingSimD3d11 simBinding = (currentSession.GraphicsBinding as GraphicsBindingSimD3d11);
+SimulationUpdate update = new SimulationUpdate();
+// Fill out camera data with current camera data
+...
+SimulationUpdate proxyUpdate = new SimulationUpdate();
+simBinding.Update(update, out proxyUpdate);
+// Is the frame data valid?
+if (proxyUpdate.frameId != 0)
+{
+    // Bind proxy render target
+    simBinding.BlitRemoteFrameToProxy();
+    // Use proxy camera data to render local content
+    ...
+    // Bind back buffer
+    simBinding.ReprojectProxy();
+}
+else
+{
+    // Bind back buffer
+    // Use current camera data to render local content
+    ...
+}
+```
+
+## Next steps
+
+* [Tutorial: Setting up a Unity project from scratch](../tutorials/unity/project-setup.md)