Mitsuba Learning in C++

[lfbuchmann]

I am trying to implement a mitsuba 3 learning cycle in C++, where I learn scene properties from actual photographs (rather than renderings). I can successfully render my initial guess, and I can load my photographs into Mitsuba Bitmaps via

mitsuba::refmitsuba::Bitmap photo = new mitsuba::Bitmap("filename.png")

however, I try unsuccessfully to populate a suitable DrJIT Tensor with the bitmap data.

Essentially, I am asking for the translation of the python function
mi.TensorXf(mi.bitmap)
into C++

Thank you!

[njroussel]

Hi @lfbuchmann

Here you go:

ref<Bitmap> bmp = (...);
size_t tensor_shape[3] = { bmp->height(), bmp->width(), bmp->channel_count() };
TensorXf tensor(bmp->data(), 3, tensor_shape);

A word of caution though, a lot of the convenience functions for automatic differentiation in Dr.Jit have only been written in Python. Writing an entire optimization in C++ will require quite a bit of work.

[lfbuchmann]

hi @njroussel ,

thanks a lot, that worked. I am aware of the work, but it would be a really fantastic tool for my use-case. I will start with it, and hopefully it will work out well.

[lfbuchmann]

Hi @njroussel ,

I thought a little bit about the problem, and I figured out an architecture that might solve my problem without having to rewrite the whole learning in C++.

If I understand Mitsuba's architecture correctly, it is essentially a Ray-Tracing front-end to Dr. Jit, which itself does the AD and optimization. However, the calculation, including the learning loop, are all contained in the megakernel, which itself is accessible (it's even written to disk).

Could I write python code that would generate the megakernel for a given architecture, or several megakernels for a few relevant architectures, and then write a separate C++ app that "simply" deploys the kernel ?

Thank you and best regards.

[njroussel]

We would like to fully support this feature of "replaying" kernels. There already has been some work towards that goal:

• Dr. Jit Kernel Caching - Semester Project drjit#35
• Dr. Jit Kernel Caching - Semester Project drjit-core#15

In theory, this is possible. However, in practice it requires a lot of small changes (see existing PRs) to work in any context/setup without too many caveats.
In your case, an optimization loop should indeed just be replaying a few kernels over and over which is much simpler and I believe is achievable without too many changes. There isn't any documentation about the DrJit internals that are involved for such a task. You would only be left with the source code.