Neural-network based SVBRDF and SDFs

[saeedhd96]

Hi,

Thanks for releasing the new version of Mitsuba!
I was wondering if in this version it is possible to use SVBRDFs in the form of neural networks, i.e. representing BRDF parameters such as albedo and roughness by a neural network that gets location as input.
Additionally I was wondering if it is possible to do ray tracing (sphere tracing) on SDFs using Mitsuba3.

[Speierers]

Hi @saeedhd96 ,

Regarding using neural networks within the rendering algorithm (e.g. as a BSDF) is something we are very interested in, although currently it isn't supported. This PR will add support for mixing pytorch and drjit AD pipelines. In theory you could use such custom AD operation to implement a neural BSDF in Mitsuba, but this will force the integrator to operate in wavefront as synchronization will be necessary at every BSDF evaluation.

Regarding sphere tracing, I suggest you take a look at this repository.

[saeedhd96]

Hello,

I see that the PR is merged. Is it now possible to transfer gradients from the mitsuba renderer to a network?

Thanks,
-Saeed

[maxfrei750]

Hi! Yes, it is. I'd start with this tutorial: https://mitsuba.readthedocs.io/en/stable/src/inverse_rendering/pytorch_mitsuba_interoperability.html

Have fun!

[wjakob]

Just FYI: you cannot call into PyTorch from within a megakernel. Wavefront-style evaluation is your only option in this case. Sébastien's patch will go a long way into making this easier to use, but it does mean that you lose all of the benefits of megakernel-style compilation (which is sort of the point of Dr.Jit/Mitsuba 3).

The main goal of his work is to make it easier to embed Mitsuba 3 into a PyTorch computation. In other words: it allows you to conveniently add neural nets before the renderer or after the renderer. It's putting neural nets inside the renderer that is difficult, because we cannot magically "merge" PyTorch/Tensorflow kernels into a megakernel being compiled by Dr.Jit.

[saeedhd96]

Thanks for the update and the tutorial. My question is, right now, the brdf parameters are defined for each object. In a neural network based brdf, we need to define the brdf parameters as a function of location on the object. Is it possible to pass such vectors as brdf params and evaluate it? Thanks, -Saeed

…

On Fri, Sep 16, 2022, 11:14 AM Max Frei ***@***.***> wrote: Hi! Yes, it is. I'd start with this tutorial: https://mitsuba.readthedocs.io/en/stable/src/inverse_rendering/pytorch_mitsuba_interoperability.html Have fun! — Reply to this email directly, view it on GitHub <#225 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQ72WNCQ5AOKSWZXLRIK23DV6SMEPANCNFSM6AAAAAAQA3LJYA> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[Speierers]

This is already what the Bitmap texture plugin does in some ways. For instance you can define your own "procedural" texture plugin that will return different value based on the location of the shading point on the geometry (e.g. UV coordinates).

As Wenzel mentioned, if you want to use a neural network to represent this function, you will be forced to either discretize/pre-compuate that function and store it in a table or switch to wavefront mode path tracing which will be considerable slower and use more VRAM.

[asprasan]

The paper Neural BRDF now integrates neural BRDF evaluation into mitsuba. While they provide a C++ plugin, it's not meant for mitsuba3 unfortunately.

But, do I understand correctly, that if I had a pre-trained MLP then I could quite easily write a python plugin for evaluating BRDF from this MLP? I suppose it should take advantage of the Dr. Jit acceleration as I do not require gradients to be passed on to the neural network.

[saeedhd96]

I wasn't able to do this. This would require to implement neural networks (with or without backward prop) in drjit, which I couldn't do because I didn't find the available matrix multiplication operations in drjit enough. In fact I think it was impossible at the current state of drjit because it only supports matrix multiplication operation for 4x4 at most (Matrix4f, correct? Also, .matmul() doesn't work on TensorXf()).

What we have implemented is in pytorch, and requires disabling the megakernel, and supports both forward/backward pass.

[njroussel]

It takes some work (it's not implemented), but definitely possible - someone shared a matrix multiplication routine with TensorXf objects here.

Anyway, looking forward to your PyTorch implementation 😄

[asprasan]

Hi @saeedhd96 , just a friendly follow-up on your implementation. Has it been released? If not, where should I keep an eye for its release? Thanks in advance.

[saeedhd96]

Hi,

I made a notebook for this specific topic, please see #718.