How do I make my viewer lighting match the lighting in AR?

[vanessauso]

I work with 3D models for augmented reality and we are having a hard time trying to make our 3D viewer parameters make the object seems closer to what it will be in the AR environment. We use blender evee to compare the 3D models to the real object but in our 3D viewer it appears to have a really strong light that make our objects seen lighter and/or too white (especially with white objects), sometimes it looks to bright in one side and in the other side of the same object, it is really dark, doesn't have a smooth transition between shadow and light, but in augmented reality it seems darker, close to blender but not so realistic.
So if we darken the model to match the viewer, it will be too dark for AR.
Does someone know the best parameters to use in a viewer so it will look closer to AR?

[elalish]

@vanessauso To some extent this is impossible by design. The AR viewers are increasingly estimating environment lighting from the camera and applying that to make the model match its surroundings, and of course any non-AR lighting therefore cannot match this.

As for tweaking the non-AR lighting, you can darken the render by lowering the value of the exposure attribute, or get completely custom lighting by applying a custom environment-image.

[NEONFIVE]

I've the same problem to match the lighting in different viewers.
If i compare lighting from Model-Viewer to Scene-Viewer and ARKIT in non AR Mode there are some differences.

1. Environment-Image did not match - is it possible to load a custom environment-image to scene-viewer? Or maybe we can ask which default Environment-image is used by Scene-Viewer?

2. There is also a dynamic top light in Scene-Viewer which i think is needed for the ground shadow but the light illuminates also the main geometry mesh.

3. AR Viewer (Non-AR Mode) on Apple Devices are also using it's own environment-image which is not customizable.

The main question is how we could match these 3 viewers so any object on any device looks almost same in NON-AR Mode?

[mrdoob]

The main question is how we could match these 3 viewers so any object on any device looks almost same in NON-AR Mode?

There is a workaround, as long as you're okay ignoring the user's environment completely.

I don't know if USDZ supports this but with GLTF has EXTENSIONS.KHR_MATERIALS_UNLIT which ensures the material ignores the scene lightning. Then you can use Blender (or whichever software you use) to bake the lighting as texture. Your model should look identical in all viewers then.

[mikkoh]

Since this might be a 3D Model used to sell a real world product it might be best to let ARQuicklook / Scene Viewer to do the light estimation and not use unlit. If light estimation is performed then the materials will be more realistic which means it will be more likely increase product confidence for the Buyer/User. If unlit is used then it may actually degrade the feeling that the AR model which you're previewing is an accurate representation of the real world product.

What could be done is to make the environment/light map for the Model Viewer match the most likely lighting that the product would be viewed in.

[mrdoob]

Privacy issues aside, I would be curious to see what would the average cubemap be if ARCore collected cubemaps from people using Scene Viewer 🤓

[vanessauso]

Thank you for the suggestions!
I compared the different environments and the google model viewer is the one that make the object seen different, in this picture I am sending, you can see Blender vs AR vs Google Model Viewer vs GLTF viewer, in this sequence, so the gltf viewer is the one closer to blender and AR viewer, what is the difference for google viewer?

[elalish]

@vanessauso Thank you for the comparison! Would you mind sharing your model and the environment images you used so I can repro? Also, when you say AR, do you mean WebXR, SceneViewer or QuickLook?

[vanessauso]

@vanessauso Thank you for the comparison! Would you mind sharing your model and the environment images you used so I can repro? Also, when you say AR, do you mean WebXR, SceneViewer or QuickLook?

Sure! I am sending you the link where the files are, you can also find the environment image (HDRI) in the same folder. Here is the link: https://drive.google.com/drive/folders/1enMvk2tZN30utxErOX5ggZ2526nBLcMK?usp=sharing

About the AR, I mean both SceneViewer (Android) and QuickLook (iOS)
Thank you for the help!

[elalish]

@vanessauso This looks to be a difference in exposure, so I'd recommend turning down the exposure attribute to less than 1. It could also be a difference in tone-mapping, but that will be more subtle. The exposure parameter will not affect SceneViewer or Quicklook. Keep in mind that sceneViewer, Quicklook, and gltf-viewer use their own lighting which you cannot adjust, so they will never match entirely. If you use webXR mode for AR, then the object will be rendered consistently between 3D and AR.

[v-prgmr]

@elalish , How can one use webXR mode for AR? I have the same problem where the object in the scene viewer has irregular light illuminations (it looks more like a lens glare).

[elalish]

@v-prgmr Can you post a screenshot of what you're seeing and tell us what device and browser and model-viewer version you're using? WebXR is the default AR mode on Chrome Android.

[v-prgmr]

@elalish I will attach a gif.

Device: Android 10 (OnePlus 6)
Browser: Chrome
Scene viewer: Version 1.0

On Model-viewer I don't see any such artifacts. However with regard to Model-viewer, when I use the webxr mode for deploying the content in AR, I experience a significant drop in performance (fps drops) when compared to using scene viewer mode for deploying the same model. Any reason why this happens??

[elalish]

@v-prgmr Yes, that's currently a limitation of webXR, but not for long. Soon we will have the same ability there to dynamically rescale the render to maintain frame rate, which is what the others are doing. Still, the fact that you're dropping frame rate implies that your model could probably benefit from optimization. You may want to look at decimating and DRACO encoding the geometry and perhaps downsampling your textures.

@tpsiaki regarding the odd lighting in SceneViewer's 3D mode.

[v-prgmr]

@tpsiaki, any update on the lighting artifact? The odd lighting artifact occurs not only in the 3D viewer but also when deployed in AR.

[v-prgmr]

@elalish @tpsiaki any update on this?