How To Best Handle Worldspace UI For VR

[TKTSWalker]

I'm working on a Stride3D project and would like to render a number of UIs onto the screen for VR. However, the setup of a render texture for each page gets extremely taxing. Is there any way to do so in a better way?

I wanted to use the UI object but that does not seem to work unfortunately.

I would be fine with using sprites and the like to create a UI system template, turn it into a prefab and open source it but i'm on a time budget to have a project completed by March.

[yuechen-li-dev]

This should be moved to discussions.

I would recommend using SkiaSharp that Stride includes for rendering UI like elements. However, Skia doesn't work in DX11 and only works either on CPU or Vulkan/DX12 GPU. I've made some tests where I've gotten some good results for render Skia elements in 3D space, however, rendering is the easy part, focus is the real issue with UI, as the game has to know that you clicked on the object and not something else.

If you're interested, I'll upload my research from this failed/stalled test project PoC to github for you guys to play with.

[TKTSWalker]

Yes please, that would help a lot!

[yuechen-li-dev]

https://github.com/yuechen-li-dev/SkiaTest2

These scripts pretty much generate a 3D in game UI screen and use Skia to render text/shapes, as well as some neat little debug stats. I made this to figure out the extent of the Silk.Net integration in Stride in the latest patch.

You need to build Stride with Vulkan enabled on Windows to get the Vulkan path working. It's possible, but I remember there being a bug that I have to bypass to get Vulkan enabled Stride to build on windows.

Add probe scripts to any entity in Game Studio and run the game and you should be able to see it working.

[Eideren]

We do have support for worldspace UI:

Set Fullscreen and Billboard off on the UIComponent

[TKTSWalker]

Hello! Any tips for getting this working in VR? No matter what I do, this doesn't seem to work (And was the first method I tried)

[TKTSWalker]

I managed to get somewhere with this; turns out Stride VR doesn't automatically render the UI even with UIComponent and Billboard.

However, I managed to get something working!

There's good news and bad news; the good news? I was able to use the demo scene in VR! The bad? I had to use only my right eye because the left would have a wrongly rendered image

I believe the way in which I managed to get rendering working is part of why this is

Specifically, I found that setting the UIRenderFeature to the opaque render stage is what did it. However, i'm at a loss and don't know how to get a proper VR image out. It's very close but the left eye's just too offset to be good.

I am extremely willing to push development on the VR end further and have started making a few things; specifically a system that uses raycasts to simulate pressing items on the page in VR. However, I am hoping for some help on the UI rendering side. I'm on a bit of a deadline and really want to use Stride3D as the VR engine for the research i'm doing!