Inverse Rendering for EM fields

[pingpongballz]

Hi all, I know this isn't my first time here 😅, but was wondering if this certain idea is feasible in the context of inverse rendering

Please pardon my lack of knowledge, I come from an electrical engineering background 😅

Basically, I have an antenna at a certain location $r$ that picks up scattered electromagnetic fields from an object.

The transmitter shoots rays, and it radiates off the surface. The intensity of the radiation as seen by the receiver is as follows.
$\vec{E_s} = 2ik\iint_{rays} \hat{r} \times (\hat{r} \times ( \hat{n} \times \hat{H_i} ) e^{ik(t - \hat{r} \cdot \vec{r'})}ds'$, where $\hat{r}$ is the unit camera direction, $\vec{r'}$ is the hit location of the rays, and $\hat{H_i}$ is the initial ray polarisation. The result is a single 3D complex vector.

From what I've read in literature, using the Reynold's transport theorem, I can possibly decompose the derivative of this integral as $\frac{\partial}{\partial \pi} \vec{E_s} = \iint \frac{\partial}{\partial \pi} \vec{F}(r', \pi) ds' + \oint \Delta{\vec{F}} (\frac{\partial \vec{r'}}{\partial \pi} \cdot \hat{n}) dl$, where $\vec{F} = 2ik\hat{r} \times (\hat{r} \times ( \hat{n} \times \hat{H_i} ) e^{ik(t - \hat{r} \cdot \vec{r'})}$.

Then I can possibly use mitsuba's direct projective integrators to perform inverse rendering in the context of this?

I tried looking into Sionna RT, but it is a completely different application use case.

If it can be done, do I have to modify the sensors to behave more like the receive antenna, or maybe modify the BSDFs? Because the scattered field emanating from the surfaces do not really behave like the reflection types given in mitsuba.

Once again, thank you for your time 😄

[njroussel]

Hi @pingpongballz

Then I can possibly use mitsuba's direct projective integrators to perform inverse rendering in the context of this?

Yes, that's the sort of integrals that we solve in inverse rendering.

If it can be done, do I have to modify the sensors to behave more like the receive antenna, or maybe modify the BSDFs? Because the scattered field emanating from the surfaces do not really behave like the reflection types given in mitsuba.

I'm not so sure about this, because I don't quite know what the differences are. I imagine that these would indeed both need to be changed to correctly reflect how the scattered field behaves. One important note is that we don't have any tools in Mitsuba to efficiently handle differentiation of polarized light.

[pingpongballz]

Hi @njroussel,

Thank you! Are there any methods or functions built into mitsuba to compute the term $\oint \Delta{\vec{F}} (\frac{\partial \vec{r'}}{\partial \pi} \cdot \hat{n}) dl$?

Once again, thank you 😀

[njroussel]

Currently, in Mitsuba, only the *_projective integrators are capable of handling discontinuities in the integrand. Most of the relevant code is in this method: https://github.com/mitsuba-renderer/mitsuba3/blob/master/src/python/python/ad/projective.py#L532. This whole logic is fairly complex, you might actually want to start at the very top of the callstack and walk your way down, i.e here: https://github.com/mitsuba-renderer/mitsuba3/blob/master/src/python/python/ad/integrators/common.py#L1013-L1053

[pingpongballz]

Hi @njroussel,

Thank you.

I was thinking for a näive first implementation, I can obtain the directly visible boundary by obtaining a mask of rays that hit or miss a scene, and then definining the boundary as the contour of the mask. From there, I just project the rays from the contour to the scene to obtain my boundary.

I then compute the boundary integral by summing up the fields on the boundary to get my boundary derivative, then add it to the interior derivative obtained by AD to get my final derivative.

Is this flow a logical flow for this type of problem? Pardon me, I'm new to this 😅

[njroussel]

I can obtain the directly visible boundary by obtaining a mask of rays that hit or miss a scene, and then definining the boundary as the contour of the mask.

I imagine this mask is an image? Depending on the scene this might work for a high resolution mask. However, more generally, you want your ray to be exactly on the shape's boundary/silhouette. If you're just using pixel boundaries you'll naturally have small bias as the pixel has a certain width. Ultimately, this would just mean that some of the terms in your boundary integral are slightly wrong.

It might still be a good approximation and be sufficient to optimize the inverse problems you're considering.

[pingpongballz]

Hi @njroussel

I see. So the flow I proposed would work in theory? Other than the boundary integral being slightly wrong from the pixel approximation?