Regularization spikes in the linear method

[davidomanovic]

The linear method minimizer has an issue where the regularization can spike from 0.001 to >1 in just one iteration which freezes the optimization even when the norm of the gradients are still on the order of ~1e-5 to ~1e-4. To deduce if the optimizer has converged, we often expect something like the grad of the norm to go to ~1e-8. As you can see, the gradient also freezes and flatlines when the regularization spikes.

I don't know if this is intentional, or due to ill-conditioning of the tangent space? Any way to mitigate this?

[kevinsung]

By default, the regularization and variation hyperparameters are chosen in each step to minimize the energy faster. You can disable this behavior by setting optimize_regularization=False or optimize_variation=False. If you want to wait for the gradient to decrease more before converging, you can also set a smaller value for the function value tolerance ftol (like zero). Also, note that the default value of the gradient tolerance gtol is 1e-5, you can set that to your liking as well.

[davidomanovic]

Yeah, I have tried that but the reg and variation tuning is definitely best. However, I am still confused as to why regularization spikes from 0.0001 to >1 in one iteration and freezes everything. Is there a way to avoid this other than hard-coding the hyperparameters? I have set gtol and ftol = 0.0 to let it run forever, but still the regularization causes the norm of the grad to freeze at ~1e-4.

[kevinsung]

There's also the optimize_kwargs argument which determines how the hyperparameters are optimized in each step as the keyword arguments to scipy.optimize.minimize. Have you tried playing with that?

[davidomanovic]

Yep, tried just by setting bounds on the regularization but it makes the minimization suuuper slow compared to default settings

[kevinsung]

Do you have an example where the linear method fails in the way you describe here, and you were able to find a lower minimum using a different optimizer (like L-BFGS-B)? If so, then could you provide a minimal code example to reproduce it?

[davidomanovic]

Do you have an example where the linear method fails in the way you describe here, and you were able to find a lower minimum using a different optimizer (like L-BFGS-B)? If so, then could you provide a minimal code example to reproduce it?

Could it possibly be multithreading? Why is my final energy different from run to run? Sometimes the differences can be like -108.720 and -108.723, which is quite significant.

[kevinsung]

Do you have an example where the linear method fails in the way you describe here, and you were able to find a lower minimum using a different optimizer (like L-BFGS-B)? If so, then could you provide a minimal code example to reproduce it?

Could it possibly be multithreading? Why is my final energy different from run to run? Sometimes the differences can be like -108.720 and -108.723, which is quite significant.

If you suspect multithreading, you should be able to disable most of it by setting the environment variables

export OMP_NUM_THREADS=1
export RAYON_NUM_THREADS=1
export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1