how it is possible for parameter prim_index within SurfaceInteraction3f to be differentiable

[qiangdadi]

I’m a newcomer to Mitsuba 3. Sometimes I wonder how it is possible for parameter prim_index within SurfaceInteraction3f to be differentiable. Please forgive my limited ability to understand. Are there any code examples or articles I can refer to? The basic idea is I want to rewrite a method to optimize the parameter using backpropagation, but I can’t think of a good way to handle the indexing.

[njroussel]

Hi @qiangdadi

The prim_index isn't differentiable, or at least it shouldn't be. What let you to believe that?
Only floating point types can be differentiated with Dr.Jit/Mitsuba, prim_index is an integer.

[qiangdadi]

Oh, This is just my own view. But I wonder how to optimize the object placement with ray tracing? Dose hit point make sense? or some related variables?

[ziyi-zhang]

Hi,

If you mean something like pose estimation (changes location and orientation but not the shape itself), a minimal example can be found here.

[qiangdadi]

Yes, I have read the example. But I want to know the process much deeper. I just confused how the relationship between ray tracing and object placement. In other words, I want to know how the AD graph be built between ray tracing (SurfaceInteraction3f ) and object placement. Just the hit_point makes the AD graph connect?

[ziyi-zhang]

I see.
Ultimately the question is "how is the pixel color connected to the location of a mesh, given that prim_index is discrete and non-differentiable?"

(1) The ray tracing operator itself is not differentiable and does not need to be.
Ray tracing with a mesh gives prim_index and barycentric uv. Both are detached.
With these values, we compute the SurfaceInteraction3f struct (si) in compute_surface_interaction(). This computation will be attached. Concretely, if the vertex_positions of this mesh has gradient enabled, all variables (like shading normal, intersection point, etc) that depends on vertex positions would also be attached.
Since the final pixel color depends on these attached variables, gradients can flow all the way from the pixel color to si and then to vertex positions.

(2) Does this always give correct gradients?
Most of the cases, yes.
The above strategy essentially considers the local sensitivity at the current hit point. This works well if we differentiate shading normals, albedo colors, BSDF parameters, etc.
However, it breaks if the change is "non-local" -- a typical example is that at the visibility silhouette of an object, the ray-tracing operation could hit the occluder or the background shape. Since the system only knows the information on one object (again, it just has prim_index and uv), the above design would not be able to handle such cases. You can find more introduction in this video.

[qiangdadi]

Thank you for the reply. It helps me a lot.