Transmission looks incorrect when no opaque object behind

[hybridherbst]

Description

Seems transmission is wrongly calculated when there's no other non-transparent object behind a transmissive object.
This can easily be seen with DragenAttenuation.glb when viewed from the side:

Compare this with Babylon or https://gltf-viewer.donmccurdy.com/:

This is also a problem in Augmented Reality: I understand that no refraction can occur (since the camera image isn't actually accessible) but right now objects are simply colorless instead of transmission-colored as expected.

[elalish]

That's interesting; can you check if that's specific to the volumetric extension? We had this working properly for the simpler transmission materials before...

[hybridherbst]

I think so, same with the transmission test model which doesn't have the volume extension:

[elalish]

@mrdoob This looks like a regression; do you have any idea what CLs we should look at?

[mrdoob]

Hmm, I've just checked previous versions in https://modelviewer-tester.glitch.me/ and I'm not sure this is a regression 🤔

[elalish]

Strange, I remember we checked this in three.js when we were adding support. I'm not sure how MV could break it, but I'll have a look. Does three.js have an example demoing this anywhere?

[mrdoob]

If I remember correctly we never managed to have color.

We managed to have some shading showing which was better than nothing:

mrdoob/three.js#22009 (comment)
mrdoob/three.js#22425
mrdoob/three.js#22473

[elalish]

Oh yeah, I guess that was as far as we got. @hybridherbst the difference between MV and those other renderers is they are not rendering to a transparent canvas and allowing that to be stacked in the browser on top of other DOM. We are, but it doesn't (can't) work as well because information is lost. Still, I'm curious if it's possible to do better...

[hybridherbst]

Oooh, interesting, so this is a general issue in three when rendering to a transparent canvas?

Would either of you be able to give a quick summary of the related effects that make this hard? I went over the linked issues but they seem to mostly discuss workaorunds and not the underlying issue.

[mrdoob]

A very basic summary is that transmission is material_color * color_behind.
When the background is transparent there is no color_behind to multiply with.

I would also not consider this a three issue. In fact, as far as I know, three is the only renderer that tries to do something about it by not assuming that the background is black.

[elalish]

The problem is blend functions. For proper transmission, you want background pixels in scene linear light, then attenuate them with color and add in scene linear light from reflections. However, when outputting to a transparent canvas, all your float linear light values (of reflection, since that's all you can calculate without knowing there's a background) get tone-mapped / clamped down to 0-255, and your background is 0-255 as well. Now you get to blend them together after the fact, but you really don't know how they relate anymore. How intense is the background "light" compared to the reflections light? Is the color in your buffer attenuation color that should be multiplied to the background, or reflected light color that should added?

[elalish]

Agreed with @mrdoob; it's a pretty cool feature that this is supported at all. It can't be actually correct, but it does a decent job for uncolored (glass-like) transmission.

[elalish]

@hybridherbst This is the line where the hack comes in to create the reflections despite no pixels behind. You can see in the PR comments that your issue was noted, but unfixed. I'm guessing adjusting this line will be the key to fixing it. There's also some interesting related discussions here. If you can find something that works better, I'll be happy to help push it through the merging process in three.js. Considering this is a pretty wild hack in the first place, I don't think there should be too much resistance to changing it.