Reflection issue in Androids Scene-viewer

[eyefun]

Description

Reflection appears not to translate adequately from model-viewer to scene-viewer
We are experiencing issues with the AR experience of 3D models holding reflections (metal like) on Android devices (scene-viewer). iOS (apple-quicklook) works fine.

Trying the URL (https://preview.eyefun.nl/test/reflections/index2.html) leads to results shown as attached.

AR view Android (screenshot): android-scene-viewer.jpg
AR view iOS (screenshot): apple-quicklook.jpg

The AR view on Android devices looks frosted, not as shiny as it should.

Because the model-viewer results are similar on Android and iOS, we think something may go wrong when scene-viewer takes over on Android devices.

Attached you'll find the glb-file, testReflection2.glb

Testing our model on https://arvr.google.com/scene-viewer-preview produces 0 faults.

Any ideas?

Live Demo

https://preview.eyefun.nl/test/reflections/index2.html

Browser Affected

• Chrome
• Edge
• Firefox
• Helios
• IE
• Safari

OS

• Android
• iOS
• Linux
• MacOS
• Windows

Versions

• model-viewer: v0.10.0
• three.js: r??

testReflection2.glb.zip

[elalish]

@eyefun This is a complicated issue. First, model-viewer, scene-viewer and quick-look all use different lighting (we cannot specify lighting to those apps, and they are generally calculating it on the fly with the camera to match your environment, and using heavy approximations).

As for shiny materials looking blurry, beyond the differences in the lighting, there is a known issue with SceneViewer's renderer, Filament, blurring shiny materials, which is especially notable in the khronos-DamagedHelmet comparison here. @tpsiaki

[judasbenhur]

We're running into the same issue with PBR materials. Is there any update to this or best practices for handling this?

[judasbenhur]

Identical issue from 4 months ago: https://stackoverflow.com/questions/60542324/google-model-viewer-no-reflections-on-object-on-android

[elalish]

@judasbenhur Have you tried using the webxr AR mode instead of SceneViewer? WebXR stays in the browser, so it uses the same renderer as the 3D view and keeps all the resources loaded; that'll be the best way to get consistency.

Thanks for pointing me to the SO question; I need to find a way to get notified of those.

[judasbenhur]

@elalish can you point me into the right direction for implementing this webxr AR mode? When you say it stays in the browser, does that mean it'll still be able to be rendered in AR?

[elalish]

@judasbenhur Indeed, WebXR is a new API for in-browser AR & VR. As long as you're using the latest (v1.0) model-viewer and Chrome 83+ on Android, it's already the default AR experience. See https://modelviewer.dev/examples/webxr.html

[romainguy]

@elalish Filament's processing of IBLs doesn't explain the difference shown here. It's more likely that the difference (since it's between ARCore and ARKit) is in the generation of the dynamic env map (ARCore provides a 16x16 cubemap, it won't be sharp no matter what).

[judasbenhur]

@romainguy thats 16x16 px I assume? That would explain a lot. Is there any plan to improve this resolution in Scene Viewer?

[romainguy]

Yes, it's 16x16px per face.

[judasbenhur]

@elalish whats the best way to verify/test that the viewer stays in the browser? I'm still seeing reflection and brightness mis-matches between model and scene viewer.

I've confirmed to be using version 1.0.0. on a Samsung Galaxy S7 with
Chrome 83.x. I've set the ar mode inside the element as follows: ar-modes="webxr". Am I missing anything else?

[elalish]

@judasbenhur Just put an HTML element as a child of model-viewer; if you're in webXR, it'll still show up when you've entered AR. I'm currently tracking down some bugs on failing to enter webXR, which you may have run into.

[vortice3D]

Hi there, @eyefun, @elalish, @judasbenhur and @romainguy:

I am precisely experiencing a problem that is related to the issue of reflections/refractions of light, although in my case on a translucent material such as glass.

I have a glass ball in GLTF/GLB that looks like this (screenshot of BJS Sandbox):

The WebGL visualization through modelviewer already shows the glass, but with a very little "presence", both in Android-Chrome, and on iOS-Safari, even taking into account that in the PBR channel of baseColor have been introduced subtle traces of dirt, to give more realism to the whole:

Of course, not being able to add an environment to the reflection in the glass is a big problem here.

Same way, when we switch to AR on Android-Sceneviewer, the glass is barely visible, ruining much of the (augmented) realism:

However, in iOS-Quicklook some magic is done and it looks really convincing and appealing:

As an important info, note that the USDZ used in iOS has been generated, by means of Reality Converter, from the GLTF/GLB used in Android, so I understand that either this software has introduced some component that makes the crystal appear or directly ARKit-Metal interprets better this specific aspect of reflectance than ARCore-Filament.

This said, any "reflection" on this subject will be welcome.

Thanks for your time.

[elalish]

@vortice3D The problem here is that glTF as is does not have a way to specify a glass material, which is the purpose of the transmission extension, which we don't support quite yet. The alpha channel of the base color is technically for partially empty surfaces, like gauze, hence the lack of reflectivity. I've seen decent results just by setting alpha to be less transparent, but it's certainly a hack and this is why different renderers treat it differently. Once everyone is using the transmission extension instead things will be much more consistent.

[romainguy]

The specific IBL used will also affect the appearance dramatically.

[vortice3D]

@vortice3D The problem here is that glTF as is does not have a way to specify a glass material, which is the purpose of the transmission extension, which we don't support quite yet. The alpha channel of the base color is technically for partially empty surfaces, like gauze, hence the lack of reflectivity. I've seen decent results just by setting alpha to be less transparent, but it's certainly a hack and this is why different renderers treat it differently. Once everyone is using the transmission extension instead things will be much more consistent.

I had already thought of adjusting the texture of the base colour channel, as a last resort.

As you say, that's a hack, but if there's no other solution we'll have to try it.

Best regards.

[vortice3D]

The specific IBL used will also affect the appearance dramatically.

Thanks for the immediate response, @romainguy.

I will report what you say to the 3D artist, in case he could do something about it, but I am not very hopeful, according to what a Khronos 3D Formats working group member is saying in this post:

"The Khronos 3D Formats working group is working on the official extension for IBL, but this will take time. We are not likely going to see many tools that have IBL in the glTF until that happens."

By the way, I suppose that ModelViewer's environment-image parameter only affects the WebGL visualization but not its AR counterpart, isn't it?