Open3D visualizer like visualization quality

[salmanmaq]

Cross-posting from the Open3D github discussions: isl-org/Open3D#4530

I am currently doing RGBD integration based reconstruction using Open3D. When I visualize the reconstructed models with Open3D, the results are great! The colors are accurate as they were in the video source. See below:

However, when I visualize using Google's model-viewer, the results are different especially the colors are not as rich and there are geometrical artifacts that do not seem to be present in the Open3D based visualization (for example above the red line in the image below). I can adjust the lightning in the model viewer a bit but the results are almost never as accurate as Open3D. Maybe experimenting more with lightning here will help, but it won't resolve the geometrical artifacts. Screenshot from model-viewer below:

I am using GLTF file extension. I would like to get results as close to the Open3D visualizer as possible and be able to display them in the browser. Any tips and suggestions on how to do that or get there? I don't know too much about computer graphics but I am willing to learn more if that helps me with this.

[elalish]

For 3D scans that just produce color (as opposed to more advanced ones that also produce roughness/metalness maps), they really aren't physical since the lighting and shadows have become mixed up with the material color. For these I would recommend using the glTF KHR_materials_unlit extension, which will simplify our shaders used in rendering and ignore applied lighting entirely.

[elalish]

In case you were still having trouble with models marked as Unlit, see #3152 for a fix.

[salmanmaq]

Hi. Thank you for your reply earlier and now. A few updates on this since then:

1. The 3D models that we get do not have a texture map
2. Our models only have vertex colors
3. Open3D uses filament as the rendering engine that is different from threejs used by model-viewer. For filament, Open3D uses some particular settings that help it render the models as in the image.
4. Our models have lighting information baked in that came from the environment in the pictures/video we recorded. We are currently not trying to separate out the lighting information.

Some questions that I have:

1. Since our models have lightning information baked in, does that mean that our models are unlit?
2. When can we expect this fix to be available via: <script type="module" src="https://unpkg.com/@google/model-viewer/dist/model-viewer.min.js"></script>

Thank you so much for the fix and for getting back!

[elalish]

1. Since our models have lightning information baked in, does that mean that our models are unlit?

Correct, that makes them unlit (you don't want to apply external light to them). If you mark them as such, that should work fine with vertex colors.

2. When can we expect this fix to be available via: <script type="module" src="https://unpkg.com/@google/model-viewer/dist/model-viewer.min.js"></script>

Maybe 6 or 8 weeks? You're welcome to build master and serve it yourself if you need it sooner.

[salmanmaq]

Thank you so much! I tried the latest master branch locally as mentioned and there is certainly some improvement.

Here is a visualization using the Open3D visualizer:

Here is a visualization using modelviewer.dev/editor:

And finally, one using the build from the latest master branch code:

Not exactly the same but I think it's a big improvement, the colors are much more accurate now. I suppose Open3D is using some lightning as well that's why we can see more shine in for example the floor tiles. But appreciate your help and guidance! I can wait till it's available via <script type="module" src="https://unpkg.com/@google/model-viewer/dist/model-viewer.min.js"></script> so that integration is easier. Thank you so much for all the help!

[salmanmaq]

For the scene I shared in my post (it's not the same file but it's the same room):

Using Open3D visualizer:

Using modelviewer.dev/editor:

Using model-viewer build from master branch:

I suppose there are still some differences in visualization. But as long as we can get more accurate colors, that's an improvement.