Clean up Volumetric Fog blending behavior

[clayjohn]

Closes: #112468, but only if merged along with godotengine/tps-demo#212

This PR finished the work that #111998 started. Prior to #111998 we were wrongly blending Volumetric Fog with the scene. In particular, by using the mix() function to blend Volumetric Fog, we were applying the fog's opacity twice. Effectively making thin fog much thinner. This had the effect of making fog much darker than it should be which is especially noticeable at the edge of Fog Volumes since it produces a dark outline (as reported in #101514)

This PR cleans up the Volumetric Fog code to be more consistent and use transmittance everywhere instead of opacity. transmittance makes the blending calculations a lot simpler, so this PR has the bonus effect of making our code slightly more efficient as well.

Warning: This is technically a compatibility breaking change. Users who previously compensated for the fact that opacity was multiplied twice will experience visual changes especially if they are using extreme values like we do in the TPS demo. Users who have default settings or denser fog will not notice much or any change.

At any rate, I think we need to go ahead with this and live with the small compatibility breakage as it will be much better in the long run to have the underlying bug fixed

[clayjohn]

Out of date comment, hidden for readability

For reference here are the correct blending equations:

diff --git a/servers/rendering/renderer_rd/shaders/environment/sky.glsl b/servers/rendering/renderer_rd/shaders/environment/sky.glsl
index 5ba1ab5b3f..55e2c538d3 100644
--- a/servers/rendering/renderer_rd/shaders/environment/sky.glsl
+++ b/servers/rendering/renderer_rd/shaders/environment/sky.glsl
@@ -281,7 +281,8 @@ void main() {
 
 	if (sky_scene_data.volumetric_fog_enabled) {
 		vec4 fog = volumetric_fog_process(uv);
-		frag_color.rgb = frag_color.rgb * (1.0 - fog.a * sky_scene_data.volumetric_fog_sky_affect) + fog.rgb * sky_scene_data.volumetric_fog_sky_affect;
+		fog.rgb = frag_color.rgb * fog.a + fog.rgb;
+		frag_color.rgb = mix(frag_color.rgb, fog.rgb, sky_scene_data.volumetric_fog_sky_affect);
 	}
 
 	if (custom_fog.a > 0.0) {
diff --git a/servers/rendering/renderer_rd/shaders/environment/volumetric_fog.glsl b/servers/rendering/renderer_rd/shaders/environment/volumetric_fog.glsl
index e6d28e33e0..3cb28d5550 100644
--- a/servers/rendering/renderer_rd/shaders/environment/volumetric_fog.glsl
+++ b/servers/rendering/renderer_rd/shaders/environment/volumetric_fog.glsl
@@ -167,7 +167,7 @@ void main() {
 	vec3 emission = vec3(0.0);
 	vec3 albedo = vec3(0.0);
 
-	float cell_depth_size = abs(view_pos.z - get_depth_at_pos(fog_cell_size.z, pos.z + 1));
+	//float cell_depth_size = abs(view_pos.z - get_depth_at_pos(fog_cell_size.z, pos.z + 1));
 
 	vec4 world = scene_params.transform * vec4(view_pos, 1.0);
 	world.xyz /= world.w;
diff --git a/servers/rendering/renderer_rd/shaders/environment/volumetric_fog_process.glsl b/servers/rendering/renderer_rd/shaders/environment/volumetric_fog_process.glsl
index 47c991292f..72cba2ce1c 100644
--- a/servers/rendering/renderer_rd/shaders/environment/volumetric_fog_process.glsl
+++ b/servers/rendering/renderer_rd/shaders/environment/volumetric_fog_process.glsl
@@ -373,7 +373,7 @@ void main() {
 	vec3 emission = vec3(emission_u >> 21, (emission_u << 11) >> 21, emission_u % 1024) / vec3(511.0, 511.0, 255.0);
 	emission += params.base_emission * params.base_density;
 
-	float cell_depth_size = abs(view_pos.z - get_depth_at_pos(fog_cell_size.z, pos.z + 1));
+	//float cell_depth_size = abs(view_pos.z - get_depth_at_pos(fog_cell_size.z, pos.z + 1));
 	//compute directional lights
 
 	if (total_density > 0.00005) {
@@ -719,7 +719,7 @@ void main() {
 
 		prev_z = z;
 
-		imageStore(fog_map, fog_pos, vec4(fog_accum.rgb, 1.0 - fog_accum.a));
+		imageStore(fog_map, fog_pos, vec4(fog_accum.rgb, fog_accum.a));
 	}
 
 #endif
diff --git a/servers/rendering/renderer_rd/shaders/forward_clustered/scene_forward_clustered.glsl b/servers/rendering/renderer_rd/shaders/forward_clustered/scene_forward_clustered.glsl
index 514fe6b416..3ccd1339c2 100644
--- a/servers/rendering/renderer_rd/shaders/forward_clustered/scene_forward_clustered.glsl
+++ b/servers/rendering/renderer_rd/shaders/forward_clustered/scene_forward_clustered.glsl
@@ -1193,7 +1193,7 @@ void fragment_shader(in SceneData scene_data) {
 	vec2 anisotropy_flow = vec2(1.0, 0.0);
 	vec3 energy_compensation = vec3(1.0);
 #ifndef FOG_DISABLED
-	vec4 fog = vec4(0.0);
+	vec4 fog = vec4(0.0, 0.0, 0.0, 1.0);
 #endif // !FOG_DISABLED
 #if defined(CUSTOM_RADIANCE_USED)
 	vec4 custom_radiance = vec4(0.0);
@@ -1435,6 +1435,8 @@ void fragment_shader(in SceneData scene_data) {
 
 	if (bool(scene_data.flags & SCENE_DATA_FLAGS_USE_FOG)) {
 		fog = fog_process(vertex);
+		fog.rgb *= fog.a;
+		fog.a = 1.0 - fog.a;
 	}
 
 	if (implementation_data.volumetric_fog_enabled) {
@@ -1446,16 +1448,16 @@ void fragment_shader(in SceneData scene_data) {
 		vec4 res = vec4(0.0);
 		if (bool(scene_data.flags & SCENE_DATA_FLAGS_USE_FOG)) {
 			//must use the full blending equation here to blend fogs
-			float sa = 1.0 - volumetric_fog.a;
-			res.a = fog.a * sa + volumetric_fog.a;
-			if (res.a > 0.0) {
-				res.rgb = (fog.rgb * fog.a * sa + volumetric_fog.rgb * volumetric_fog.a) / res.a;
-			}
+			res.a = fog.a * volumetric_fog.a;
+			res.rgb = fog.rgb * volumetric_fog.a + volumetric_fog.rgb;
 		} else {
 			res = volumetric_fog;
 		}
 		fog = res;
 	}
+#else
+	fog.rgb *= fog.a;
+	fog.a = 1.0 - fog.a;
 #endif //!CUSTOM_FOG_USED
 
 	uint fog_rg = packHalf2x16(fog.rg);
@@ -2886,8 +2888,8 @@ void fragment_shader(in SceneData scene_data) {
 #endif
 
 #ifndef FOG_DISABLED
-	diffuse_buffer.rgb = mix(diffuse_buffer.rgb, fog.rgb, fog.a);
-	specular_buffer.rgb = mix(specular_buffer.rgb, vec3(0.0), fog.a);
+	diffuse_buffer.rgb = diffuse_buffer.rgb * fog.a + fog.rgb;
+	specular_buffer.rgb = specular_buffer.rgb * fog.a;
 #endif //!FOG_DISABLED
 
 #else //MODE_SEPARATE_SPECULAR
@@ -2902,8 +2904,7 @@ void fragment_shader(in SceneData scene_data) {
 #endif //USE_NO_SHADING
 
 #ifndef FOG_DISABLED
-	// Draw "fixed" fog before volumetric fog to ensure volumetric fog can appear in front of the sky.
-	frag_color.rgb = mix(frag_color.rgb, fog.rgb, fog.a);
+	frag_color.rgb = frag_color.rgb * fog.a + fog.rgb;
 #endif //!FOG_DISABLED
 
 #endif //MODE_SEPARATE_SPECULAR

However, this results in volumetric fog looking extremely strong, just like in #112468

[Calinou]

Tested locally, it works as expected. Code looks good to me.

Testing project: test_volumetric_fog_blend.zip

The test project showcases volumetric fog with very low density (0.001), lights with high volumetric fog energy (200.0) and VoxelGI affecting a FogVolume with density 0.5.

4.5.1.stable	master	This PR

In this example, the PR only makes a tiny visual difference compared to master.

I've opened a documentation PR to go along with this change:

• godotengine/godot-docs#11457

[Repiteo]

Thanks!