Differentiable Silhouette Integrator

[GuangyanCai]

Hi there,

The aov integrator has a differentiable depth integrator and I was trying to modify it into a differentiable silhouette integrator but failed. A silhouette integrator should behave similarly to a depth integrator except it doesn't return the depth but 1 when a ray hit an object. Can you give me an idea of how to implement it? Or is there already an implementation that I didn't find?

Currently, I have complicated workaround, which set the object's albedo to 0, remove all the light sources and set the object emission to 1. The result will look the same as that of a silhouette integrator and return the correct gradient. However, it doesn't fit into our pipeline well.

Thanks in advance.

[ziyi-zhang]

Hi @GuangyanCai,

case Type::Depth:
                    *aovs++ = dr::select(si.is_valid(), si.t, 0.f);
                    break;

You can check if the depth is zero "dr.eq(depth, 0)" to tell if it hits any object. But from this definition the outcome is always binary and thus non-differentiable. Could you elaborate more about the "differentiable silhouette integrator"?

[GuangyanCai]

Right, I tried that but it's not differentiable. I also tried si.t / si.t, which gives the correct silhouette but the gradient is wrong. The silhouette integrator is mainly for computing mask loss and provides a strong regularization. Although it's binary, we should still be able to compute the gradient. The workaround I mentioned is a working solution but it's difficult to use in practice.

[ziyi-zhang]

mask loss and provides a strong regularization

I see. It's for the shape mask loss.
This is very common in CV but is a little bit hard in CG, especially in a PBR renderer.

The correct way to do it in my mind is roughly as follows:
Render an aov channel of hit or not, compare with reference mask to know for every pixel should we see more or less of the shape. Note that this information will be very sparse, only non-zero on the visibility silhouettes.
To use the derivative, we cannot rely on autodiff and have to explicitly sample the silhouettes to compute some "boundary derivative". You can take a look at the "sppp" part of the logic in the projective integrator, which does roughly the same thing.

This cannot be achieved by differentiating si.t. Intuitively, the mask loss should encourage the shape to expand or shrink along the direction perpendicular to the camera ray. Whereas differentiating si.t can only propagate derivatives that changes the geometry along the camera ray, which does not change the binary occupancy rendering (what you referred to as "silhouette integrator" rendering).