Coupling a differentiable parametric geometry

[romp13]

Hi there,

Given a parametric model of a mesh (e.g. skinned mesh or blendshape model), is it possible to plug it in with Mitsuba 3 to optimize the parameters (e.g. skeleton joints or blendshape weights) based on some target renderings?
Our model is differentiable but I guess it does not help since it is not written with Dr.Jit.

I would love to hear your thoughts on how this can be done.

Thanks!

[Speierers]

Hi @romp13 ,

Indeed you will need to re-implement your skinned mesh or blendshape model in Dr.Jit in order to incorporate it within the optimization loop.

All of this can be done directly in Python. For instance you could load a couple meshes, apply some transformations on them resulting in the blended mesh that will be used for rendering. Just make sure those transformation are implemented in a differentiable way using Dr.Jit.