Compressing meshes with KTX_texture_basisu causes meshes to appear dark on Chromium on Android

[JeS24]

Description

In Chromium and Chromium-based browsers on Android, <model-viewer> displays meshes as completely dark or rarely with light and dark patches. This happens for meshes optimized / compressed using meshoptimizer, with the following command: gltfpack -i input.glb -o output.glb -cc -tc. Here, cc denotes compression level, while tc is used to "convert all textures to KTX2 with BasisU supercompression". The purpose of this compression is to save storage space and bandwidth while serving the models over the web. Oddly enough, this problem never occurs on Chromium and Chromium-based browsers on the desktop on any platform. Safari (iOS + iPadOS / macOS) & Firefox (Android, desktop etc.) are completely unaffected too.

If I remove the -tc flag, the models are loaded correctly. I have tried using https://modelviewer.dev/editor/ to check for errors. There are no errors in any case, but there are some warnings that seem relevant. These are the outputs with & without KTX_texture_basisu (-tc) and also, for the uncompressed mesh:

• Flags: -cc -tc / Max compression / KTX_texture_basisu | Size: 6 MB | Shows up dark on Chromium on Android.
  

• Flags: -cc / Medium compression / No KTX_texture_basisu | Size: 27 MB | Loads fine everywhere, but AR button may not work.
  

• No compression: Size: 34 MB | Loads fine everywhere & AR works without problems.
  

• Note that, in these images, the models look okay, because the screenshots are from Edge on PC. Check the glitch.me link below for a demo.

• As you can tell from the file-sizes, meshopt's compression is quite good. And I need this high compression level, since the actual meshes are going to be 1 - 2 GB in size. With meshopt's compression, using those flags, I have been able to bring the size down to ~300 - 400 MB, without a drastic loss in quality. Is there a workaround for this issue?

• I am also fine with another model-viewer compatible compression scheme, that can nearly match meshopt. I have tried draco, but the results haven't been as good, at least not without substantial loss in quality.

• I am also puzzled by the models loading fine on Chromium etc. on PC. Could this be due to how Chromium
  / model-viewer interacts with meshoptimizer/meshopt_decoder.js?

• Also, not sure if this is relevant, but the models have been generated via AliceVision Meshroom's photogrammetry pipeline.

Some possibly non-essential information:

1. The "Deploy Mobile" option on the editor page doesn't quite work. The model seems to load initially on the phone, but then the loading gets stuck. Refreshing doesn't help either. This always happens for models with KTX_texture_basisu in my testing.

2. This issue also seemingly affects model-viewer's AR button's functionality, but I want to test it more, since the results I have are inconsistent. At the moment, AR never works with the -tc flag applied, but it does work sometimes without the flag. By "never works", I mean, on Android, it complains about there being "something wrong with the object" and on iOS / iPadOS, the button disappears, but the QuickLook view never opens. And by "work sometimes", I mean, the same model (-cc, no -tc) without any change to any part of the process, except a different time of day, loads without issues. Just including this for completeness' sake. More testing is needed.

Live Demo

https://sparkling-mercurial-muenster.glitch.me/ (Ignore the astronaut poster)

Version

• model-viewer: v3.0.2

Browser Affected

• Chrome, version: 111.0.5563.116 (Android); 111.0.5563.147 (Windows)
• Edge Dev: 113.0.1758.0 (Android); 112.0.1722.23 (Windows)
• Firefox: 111.1.1 (Android) ; 111.0.1 (Windows)
• Chromium Dev: 114.0.5693.0 (Android); 114.0.5693.0 (Windows)
• Safari :16
• IE

OS

• Android: Major versions 10, 11, 12, 13
• iOS: 16.4
• iPadOS: 16.4
• Linux: arch, Ubuntu 22.04
• MacOS: 13.3
• Windows: 10, 11

AR

• WebXR
• SceneViewer
• QuickLook

[elalish]

Thanks for the detailed analysis! I believe the warnings are spurious - glTF validator just doesn't know about KTX2 (might file a feature request on them). AR problems aren't too surprising - for iOS, USDZ autogeneration is failing (not too surprising), so you can use ios-src until that's fixed. I think Scene Viewer doesn't yet support KTX2, so I'd only expect WebXR mode to work, but it'll have the same problem as 3D mode.

As for why Android is getting the wrong pixel values, that sounds like a problem with the basis-universal transcoder, so we'll have to follow up with them.

[elalish]

The JS console on Android has some pretty serious warnings:

WebGL-0xe08cae00]GL ERROR :GL_INVALID_VALUE : glTexStorage2D: dimensions out of range
14[.WebGL-0xe08cae00]GL ERROR :GL_INVALID_VALUE : glCompressedTexSubImage2D: dimensions out of range
sparkling-mercurial-muenster.glitch.me/:1 [.WebGL-0xe08cae00]RENDER WARNING: texture bound to texture unit 2 is not renderable. It might be non-power-of-2 or have incompatible texture filtering (maybe)?
76[.WebGL-0xe08cae00]RENDER WARNING: texture bound to texture unit 1 is not renderable. It might be non-power-of-2 or have incompatible texture filtering (maybe)?
model-viewer.js:42241 THREE.GLTFLoader: Couldn't load texture blob:https://sparkling-mercurial-muenster.glitch.me/cfdcb722-97b1-435e-90d3-40a611d55644
(anonymous) @ model-viewer.js:42241
model-viewer.js:22787 TH

This is on my Pixel 4a, and it's not showing a texture for any of your three examples. Do these have insanely large texture dimensions? I would not expect a 100MB model to render on any mobile GPU - you'll need to make a much downsampled version. And based on the size of the screen you're rendering on, that high res data would be wasted anyway.

[JeS24]

Sorry, I saw the reply but was too busy to respond.

it's not showing a texture for any of your three examples

This seems to be due to glitch.me changing CDN links, after I posted the issue or maybe the project had timed out. I have refreshed the project now and updated the src links. https://sparkling-mercurial-muenster.glitch.me/ should load all meshes now. Also...

Do these have insanely large texture dimensions?

Oh yes. These did have large textures. Meshroom defaults to 8192x8192 with 2x downscaling. So, the textures were in 4K. I have rerun the Meshroom pipeline and now, the textures for the meshes on the glitch.me site, are in 2K (4096 + 2x DS). This seems to be directly related to the "dark mesh" issue. After using a smaller texture size, <model-viewer> is loading all 3 meshes without issues, regardless of compression level. AR is working fine too. JS console on Android is also not raising any red flags.

This raises the following questions though:

1. Is this just a VRAM limitation, as in, for meshes with basisu encoding, <model-viewer> doesn't have enough memory on Android phones to decode the mesh? As far as I can recall, the Android phones that were tested, were Galaxy S20, OnePlus 7, Redmi Note 7 Pro, Redmi Note 8 Pro & Realme 3 Pro. I'd think that at least, the Galaxy & the OnePlus should have had sufficient memory.
2. This issue does not affect Firefox on Android or Safari on iPhones, which is inconsistent in relation to point 1. Any clue as to why?
   1. Is it because <model-viewer> does not have to care about AR on Firefox, since it's not supported there and so, scene-viewer doesn't take up any memory?
   2. As for iPhones, perhaps they have sufficient VRAM. Although, I have seen that, in cases, where the model is too large, the model appears plain white on iPhones / iPads, which just means that the textures are not being applied - a telltale sign of insufficient memory. They never appear dark.

I would not expect a 100MB model to render on any mobile GPU - you'll need to make a much downsampled version.

The 300 - 400 MB figure was just to highlight how good meshopt is and why I require it in my project. The actual meshes would undergo further decimation / simplification.

[JeS24]

Oh, and one more thing - if <model-viewer> fails to load any model successfully (say, darkened or white with no textures), the subsequent models also do not load properly (white, no textures) in other <model-viewer> instances on the same page. This does not affect my use-case. Just thought I'd add it here.

[elalish]

Thanks for the details! I'm honestly not sure about answers, but everything to me is pointing to GPU OOM, in which case there's probably not much actionable for us here. It doesn't surprise me that it would affect subsequent models, since all the RAM is shared. For performance, you really don't want to be anywhere near the RAM limit, so I'd recommend you push through your "further decimation / simplification" and then you shouldn't have to worry about this anymore.

[JeS24]

Fine. I will be testing out some actual models soon. Will update here once that is done.