Is there a better way to do AD through a visibility mask

[FMRobS]

Hi all,

Thank you for developing this wonderful tool.
I am trying to do something, that I believe should be possible with the reparametrized ray tracing.
Simply put, I want to compute gradients through a point to point visibility mask. Where the rays are starting from a single viewpoint and traced the an object point set.

I have the following snippet I want to make differentiable:

view_point = mi.Point3f(0,0,0) # camera viewpoint
rays = surface_interactions.spawn_ray_to(view_point) # surface_interactions: my point cloud
coverage = dr.count(scene.ray_test(rays)) / dr.shape(rays.d)[1]

However it contains a few steps that break the AD, 'ray_test' and 'dr.count' are not AD compatible.
I would like to have both forward and backward AD.
For backward AD: gradient of coverage wrt position view_point
For forward AD: for each point the derivative of the visibility mask (what comes out of ray_test) wrt coordinates of view_point

For forward AD I have the following working:

param = mi.Float(5.0)
dr.enable_grad(param)
view_point = mi.Point3f(param,5,15)
ray_reparam = surface_interactions.spawn_ray_to(view_point)
rng = mi.PCG32()
scene_params = mi.traverse(scene)
ray_reparam.d, det = mi.ad.reparameterize_ray(scene,
                                               rng, scene_params, ray_reparam,
                                              num_rays=15,
                                              kappa=1e4,
                                              exponent=3.0,
                                              antithetic=True,
                                              unroll=False,
                                              active=True)

si = dobject.scene_mitsuba.ray_intersect(ray_reparam)
mask = dr.exp(-100.0/si.t)  # 'Visible' will have t = inf. Differentiable version of si.t == dr.inf
dr.forward(param)

grad = dr.grad(mask)

Right now the gradients are still dependent on the hit length 't'.
Is there a better way of calculating the gradients (forward & backward) of a visibility mask?

[njroussel]

Hi @FMRobS

I'm not quite sure this is possible, if I understood correctly.

Fundamentally, this is a discrete & discontinuous problem. An infinitely small displacement of your camera viewpoint or of the points in your point cloud will not produce any changes in the coverage. The coverage is discrete and changes discontinuously, hence your gradient should always be 0.
As you've pointed out in your example, "at best" you can attach the distance t. But the gradient of t is not representative of any change in visibility: moving an object further away doesn't always mean that it less/more likely to be seen.

[FMRobS]

Hello Nicolas,

Thank you for your quick reply. I was hoping this was the same kind of discontinuity the reparametrization papers aim to deal with. Is there a fundamental difference I am overseeing?
Maybe I should reformulate the problem closer to light transport. Would the following work?:
I place a black sphere around a white object. The rays now either hit white or black and the trivial radiance of 1.0 or 0.0 is indicative of the visibility.
What are your thoughts?

[njroussel]

It depends on what you're measuring. If you're measuring the accumulated radiance per pixel, then yes it would work. The difference is that the measure is continuous here, whereas is in your prior example it is discrete.

Consider a single pixel that is partially seeing a black circle on a white background. Any slight translation of the black circle will slightly change the accumulated radiance of that pixel. There is a continuous change. We are basically measuring the surface area of the circle projected onto that pixel. (The way in which this is computed, with Monte Carlo methods, needs special handling of a discontinuity that pops up and we use reparametrizations for that).

However, for an arbitrary set of fixed points, the total count of how many of them are visible from some viewpoint is discrete. This number is naturally an integer. The gradient would then be either 0 in most cases or infinite in the rare cases where a point is perfectly aligned the edge of some occluder.

[FMRobS]

Thank you,
I admit, I do not (yet) understand all details of the warp field paper and the code in '_ReparameterizeOp'.
I see how that is different. On the other side, I can't wrap my head around this:
If instead of a point cloud, I consider the small patches of the surface of the object and calculate the visibility per patch. Then the 'pixels' are an area discretization of the surface. If we continue with MC sampling this visibility integral, i.e. the percentage of shadow in the surface patch, with 1 sample per patch, I believe it is the same setup as before.

Do I need more samples per patch for the reparametrization magic to start working?

[njroussel]

If you consider each point as a smallish disk, I think that should work.
I'm by no means a "reparam" expert, I might be forgetting some important details. But intuitively, I believe there is a requirement for there to be some non-zero surface area to trace rays onto rather than a single point. Such that you will indeed be measuring the:

percentage of shadow in the surface patch

This will be a continuous measure 👍