how to realize differentiable camera in mitsuba3?

[gleefeng]

I want to realize the differetiable camera in mitsuba3
for example，I want to optimize 'ThinLensCamera.aperture_radius' this key
so I just to change the code

void traverse(TraversalCallback *callback) override { Base::traverse(callback); callback->put_parameter("aperture_radius", m_aperture_radius, +ParamFlags::Differentiable);

ParamFlags::NonDifferentiable ->ParamFlags::Differentiable

after this , run the optimizer in python test environment.
opt = mi.ad.Adam(lr=0.05)
opt[key] = params[key]
params.update(opt)

but get the error :
Exception: The contribution computed by the differential rendering phase is not attached to the AD graph! Raising an exception since this is usually indicative of a bug (for example, you may have forgotten to call dr.enable_grad(..) on one of the scene parameters, or you may be trying to optimize a parameter that does not generate derivatives in detached PRB.)
drjit-autodiff: variable leak detected (7 variables remain in use)!

of course ,I don't think it's that simple, but I want to know what else needs to be modified? Is there any documentation we can refer to?

[Speierers]

Hi @gleefeng,

This sounds very related to this PR. I suggest you pull and compile this branch and run your script, hopefully, it should just work. This will soon be merged on master and available in the upcoming release.

[gleefeng]

sensor_to_world_test.zip
here is my test code, please check .

[Speierers]

I see two problems:

1. When differentiating camera parameters, it is important to use an integrator that can handle discontinuities (moving a camera will create such discontinuities). Switching from prb to prb_reparam should solve this issue.
2. The thinlens sensor doesn't implement sample_direction() which is used in the prb_reparam integrator. I will try to add it later today.

[gleefeng]

thank you！ very appreciate for your quick reply，mitsuba3 is very amazing work，I want to build a camera simulation enviroment based on it,hope that there will be some opportunities to make some contributions

[gleefeng]

Hi，@Speierers I found you had updated the code，which one branch should I use to test？ master or diff_camera?

[Speierers]

It's been merged on master so you should use master from now on.