Optimization of .to_world is not working

[AntonioDOrazio]

I have an emitter attached to a mesh sphere, and I'm trying to optimize the X,Y,Z position of the sphere, to apply a translation to the to_world attribute, given a target renderer

This is my reference

And this is my starting point

Obtained with

params["Sphere.to_world"] = mi.Transform4f.translate([10, 12, -6])
params.update()

I am using the following code:

lr = 0.5
iterations = 15
spp=16
key = "Sphere.to_world"
opt=mi.ad.Adam(lr=lr)

opt["x"] = mi.Float(transform.translation()[0])
opt["y"] = mi.Float(transform.translation()[1])
opt["z"] = mi.Float(transform.translation()[2])

errors = []

for it in range(iterations):
    params[key] = mi.Transform4f.translate(mi.Point3f(opt["x"], opt["y"], opt["z"]))
    params.update()

    image = mi.render(scene, params, spp=spp, seed=it)
    # Evaluate the objective function from the current rendered image
    loss = dr.mean(dr.sqr(image - reference_render))
    
    dr.backward(loss)

    # Optimizer: take a gradient descent step
    opt.step()
    # Update the scene state to the new optimized values
    params.update()

    # Track the difference between the current color and the true value
    err_ref = dr.sum(loss)
    print(f"\nIteration {it:02d}: parameter error = {err_ref[0]:6f}. Opt: = {[opt['x'], opt['y'], opt['z']]}", end='\r')
    errors.append(err_ref)
params[key] = mi.Transform4f.translate(mi.Point3f(opt["x"], opt["y"], opt["z"]))
params.update()
print('\nOptimization complete.')

However, the position stays the same:

Am I doing something wrong? Or am I trying to do something which is unsupported?

Thank you

[mcrescas]

Hi 👋 ,

This line params.update() should be params.update(opt). If you don't add the parameter, you are not modifying/propagating the parameters (the changes exists inside the optimizer but not in the scene).

Regards

[AntonioDOrazio]

Hi, thanks for your reply.
I edited the code to include your fix, however I still got the same result.
I also tried to remove the second update (after the .step()), as I saw here, in order to keep only the one before the render, but the values are still not optimized.

lr = 0.5
iterations = 15
spp=64
key = "Sphere.to_world"
opt=mi.ad.Adam(lr=lr)

opt["x"] = mi.Float(transform.translation()[0])
opt["y"] = mi.Float(transform.translation()[1])
opt["z"] = mi.Float(transform.translation()[2])

errors = []

for it in range(iterations):
    params[key] = mi.Transform4f.translate(mi.Point3f(opt["x"], opt["y"], opt["z"]))
    params.update(opt)

    image = mi.render(scene, params, spp=spp, seed=it)
    # Evaluate the objective function from the current rendered image
    loss = dr.mean(dr.sqr(image - reference_render))
    
    dr.backward(loss)

    # Optimizer: take a gradient descent step
    opt.step()
    # params.update(opt)

    # Track the difference between the current color and the true value
    err_ref = dr.sum(loss)
    print(f"\nIteration {it:02d}: parameter error = {err_ref[0]:6f}. Opt: = {[opt['x'], opt['y'], opt['z']]}", end='\r')
    errors.append(err_ref)
params[key] = mi.Transform4f.translate(mi.Point3f(opt["x"], opt["y"], opt["z"]))
params.update(opt)
print('\nOptimization complete.')

Is there anything else that could be wrong?

Thank you

[njroussel]

After the dr.backward() call, but before the optimizer step (opt.step()), you can try printing the gradient of your parameter by usind dr.grad(params[key]).
This should at least narrow down the problem: are the gradient zero or are the steps not applied.