realize a differentiable realistic camera

[JiangGuotai]

I want to realize a differentiable realistic camera, which contains multiple piece of lens, and I want the radiuis of the lens can be diffferentiable.
There may be ways to realize this, i am trying to realize it as follows:
Model the lens as object in the scene. However, the ray transfer process may be tricky, so are there some advice or existed tremplate to establish len groups?

[Speierers]

Hi @JiangGuotai ,

I would suggest reading carefully through this chapter of PBRT as it covers such camera models.

Making it differentiable might be difficult if parts of this model make the evaluation of a light path discontinuous w.r.t. model's parameters.

You should also take a look at this paper.

[JiangGuotai]

Hi @Speierers ,
Thank you for answering my question.
As you mentioned, what we do is to repeat the experiments mentions in the paper wrote by Sun. et al. And making the realistic camera differentiable can be tricky(most difficult part may be discontinus). We have read through pbrt, but make it differentiable still remains not easy.
So, are there some essential references during the realization of mitsuba3? [e.g: make discontinus variables differentiable].
Sorry to bother you and thank you again for the reply!

[Speierers]

If you are interested in better understanding the implementation of the integrators that deal with those discontinuities in Mitsuba 3 (e.g. prb_reparam), I would recommend you read through the following research papers:

https://people.csail.mit.edu/sbangaru/projects/was-2020/
https://rgl.epfl.ch/publications/Zeltner2021MonteCarlo
https://rgl.epfl.ch/publications/Vicini2022SDF

[JiangGuotai]

Thank you for your time. I will try it in the coming days.

[diiigle]

Just stumbled over this implementation of a multi-lens camera in appleseed. Might help you get started.
appleseedhq/appleseed#2914
Of course this is not differentiable yet which is probably the hard part to get right.

[JiangGuotai]

Thank you!
This will help a lot!