Optimizing the position of a sphere

[darabos]

As a simple experiment I want to use gradient optimization to find the best position of a sphere.

Looking at the scene params I have s.to_world, a Transform4f, and s.bsdf.reflectance.value, a Float.

If I try to follow the Gradient-based optimization tutorial, it fails if I use s.to_world instead of s.bsdf.reflectance.value:

Exception: Optimizer.__setitem__(): value should be differentiable!

If I try to follow the Object pose estimation tutorial, I can't figure out how to apply the transformation. A sphere doesn't have vertex_positions that I could transform.

(I've also asked this at https://stackoverflow.com/questions/73156122/optimize-a-spheres-position-with-mistuba-3.)

[ziyi-zhang]

Hi, I am afraid that we cannot differentiate a sphere with the current implementation.
Currently the m_to_world is non-differentiable, and in Sphere.cpp::update() we update the center and radius in-place in a non-differentiable way.

It is definitely a useful feature to have though. We can add it to future feature requests. But most of the team won't be available until the end of August =(
Without modifying the cpp implementation, the best workaround I have in mind is to use mesh spheres, at the cost of lower precision and more overhead. One can optimize their centers and radii as latent variables.

[darabos]

That's fine, thanks! I'll use some meshes.

[anderslanglands]

Can I add a +1 on this please? There's a couple of applications where I could really use being able to optimize the transforms of objects and cameras.

[coult099]

Has there been an update to this? It would be helpful in some applications I am working on to be able to take gradients with respect to an object's position.

[njroussel]

@coult099 All analytic shape primitives (sphere, disk, cylinder, rectangle) are differentiable since version 3.10,

[njroussel]

Shape primitives (sphere, disk, cylinder, rectangle) are differentiable since version 3.10,

[gioarma]

Hello, I'm using version 3.4.0, but I'm still getting this error when trying to optimize to_world of a rectangle object.
This is my code:

import mitsuba as mi
mi.set_variant('cuda_ad_rgb')

scene_dict = {
    'type': 'scene',
    'integrator': {
        'type': 'prb'
    },
    'rect':{
        'type': 'rectangle'
    }
}

scene = mi.load_dict(scene_dict)
params = mi.traverse(scene)

opt = mi.ad.Adam(lr=0.05)
opt['rect.to_world'] = params['rect.to_world']

I get: Exception: Optimizer.__setitem__(): value should be differentiable!

I don't see why, since printing the params variable I see to_world is differentiable:

[njroussel]

Hi @gioarma

This is an unfortunate difference between the SceneParameters and the Optimizer. The former can handle Transform4f types whereas the latter cannot.

In order to achieve the same goal, you can either use latent variable(s) to represent the Transform4f in the Optimizer or use the underlying matrix of the transform: my_transform4f_object.matrix (IIRC, the Optimizer can handle Matrix types).