Bump texture optimization

[gioarma]

Hello, I'm trying to optimize a bump texture, but I'm not getting the desired result, so I'm asking for a little help.
You can find the scene dictionary in the details at the bottom.
Here is my code:

Reference texture and image

My reference texture is a sum of two 2-D gaussians. The image is a simple metallic cube with the texture applied to it. The camera looks at just one of the faces, which is illuminated by a single rectangular top light.

def gauss_2d(mu, sigma, A):
    x, y = dr.meshgrid(dr.linspace(dr.cuda.Float,-1,1,1000), dr.linspace(dr.cuda.Float,-1,1,1000))
    x = dr.cuda.TensorXf(x, shape=(1000,1000,1))
    y = dr.cuda.TensorXf(y, shape=(1000,1000,1))
    return A * dr.exp(-( (dr.sqr(x-mu) + dr.sqr(y-mu)) / ( 2.0 * sigma**2 ) ) )

ref_texture = gauss_2d(0, 0.2, 0.01) + gauss_2d(0.2, 0.2, 0.01)

scene_dict = {
    'type': 'scene',
    'integrator': {'type': 'direct_reparam'},
    'cube': {
        'type': 'cube',
        'to_world': T.rotate([0,0,1], -90),
        'bsdf':{
            'type': 'bumpmap',
            'bump_texture': {
                'type': 'bitmap',
                'raw': True,
                'bitmap': mi.Bitmap(ref_texture),
                'wrap_mode': 'mirror',
                #'to_uv': mi.scalar_rgb.ScalarTransform4f.rotate([1,0,0], 180)
            },
            'bsdf': {
                'type': 'roughconductor',
                'alpha': 0.3
            },
            'scale': 10
        }
    },
    'light':{...},
    'camera': {...}

ref_scene = mi.load_dict(scene_dict)
ref_params = mi.traverse(ref_scene)

Texture modification

Now I modify the texture, setting it as a single gaussian. I tried setting it to all zeros, or all 1s, but the result is always similar.

scene = mi.load_dict(scene_dict)
params = mi.traverse(scene)
init_texture = gauss_2d(0, 0.15, 0.01) 
params['cube.bsdf.nested_texture.data'] = init_texture
params.update();

Optimization

Finally, I run the optimization. The result, however, does not converge at all. No matter how many iterations I run, the optimization gets stuck after a few iterations. Do you have any suggestions on what I should change in my code to make it work? Thank you!

opt = mi.ad.Adam(lr=0.005)
opt['cube.bsdf.nested_texture.data'] = params['cube.bsdf.nested_texture.data']
params.update(opt);

iterations = 16
spp = 50
loss_values = []
textures = []
for it in range(iterations):
    img = mi.render(scene, params=params, spp=spp)
    textures.append(np.array(params['cube.bsdf.nested_texture.data']))
    loss_ = dr.mean(dr.sqr(ref_img - img))
    dr.backward(loss_)
    opt.step()
    opt['cube.bsdf.nested_texture.data'] = dr.clamp(opt['cube.bsdf.nested_texture.data'], 0, 1)
    params.update(opt)
    loss_values.append(loss_)
    print(f"Iteration {it:02d}: error={loss_[0]:6f}", end='\r')
opt_texture = np.array(params['cube.bsdf.nested_texture.data'])

Here is the evolution of the textures in the first 16 steps:

[njroussel]

Hi @gioarma

My best guess for what is happening here is that during the optimization the bump map texture is reaching values that create a back-facing shading normal hence the black output. Once that state has been reached there is no exiting it as any infinitely-small change of the normal would not change the rendered output (the forward gradient of the normal is 0).

A much smaller learning rate should help, I believe. Any strategy that will mitigate or recover from this back-facing normal configuration should do the trick.

[gioarma]

You're right, decreasing the learning rate by 2-3 orders of magnitude already gives a good result!
I thought it was already small enough, but I was wrong

Thank you very much @njroussel