Optimizable parametric noise texture

[maxfrei750]

I’d like to implement a parametric noise texture, whose parameters (scale, strength, clipping) I'd like to optimize based on a reference image (just like the wall color in the basic inverse rendering tutorial).

I think that the First steps tutorial of drjit might be a good starting point. However, before I invest time into figuring out a solution that might be suboptimal in the end (or not work at all), I thought it might be wise to ask for a few hints and good starting points, especially regarding the differentiability of the noise parameters.

Thanks!

[njroussel]

Hi @maxfrei750

If you're generating a bitmap texture from your parameters, you'll want to take a look at this tutorial. You would be optimizing a parameter which is not part of your scene directly, but rather is used to generate/modify a scene parameters. (We call such parameters latent variables).
If you stick to mitsuba/drjit operations you're pretty much guaranteed to keep the gradient tracking throughout the whole computation of the texture.

Typically you'll also want to take a coarse-to-fine approach if that's even possible with your parametric texture. See this tutorial where we upsample the volume during the optimization.

[maxfrei750]

Thanks a ton! So to summarize: I need to take a differentiable scene parameter, e.g. the data attribute of a texture of a shader, and manipulate it using only mitsuba and/or drjit operations, to make it dependent on a latent variable, while still keeping the gradient tracking.

When looking at numpy implementations of the Perlin noise, all operations should be covered by drjit, except for np.dstack. However, I think that function should be either easy to implement or I can work around it using @wrap_ad and torch.dstack.

Generating a noise obviously involves the generation of random values (see e.g. here). I assume that I should keep these constant (i.e. not re-roll them), to maintain differentiability, correct?

On another note, while browsing the mitsuba documentation, I found Texture2f.eval_bicubic, which might be helpful for my endeavor. Does this function also keep the gradient tracking? And do I have to take special measures (e.g. setting force_drjit=True)?

[njroussel]

Exactly!

I haven't read their implementation carefully, but if you're just calling np.dstasck(arr1, arr2) that's equivalent to mi.Array2f(arr1, arr2). If possible, try to avoid @wrap_ad: there's a cost to it.

I hadn't thought about the randomness. I think it's worth trying both options here. If your texture appearance is strongly related to the used random numbers rather than the latent variables it will be hard to optimize the latter.

This is fairly undocumented territory. You should read the c++ source documentation, the python bindings are quite similar. The eval_cubic does keep gradient tracking, and the force_drjit/force_nonaccel are flags to specify if you wish to not use hardware accelerated texture operations. This texture class is what is used in the drjit tutorial you linked earlier (by the way, you might want to take a look at this issue for more documentation).