-
Notifications
You must be signed in to change notification settings - Fork 0
Description
The sqt.fit.grad.line_search function does not work properly on some (most? all?) inputs as described in the docstring:
This method is currently not working correctly and systematically returns a value of
1e-5that has been empirically determined as sufficiently small for the optimisation problems I had. This value might be too large for other optimisation problems.
Some observations on this method: it seems like the line search does not work because there are cases where the projection makes the actual descent direction in the opposite direction of the gradient. In other words, ifdis the descent direction, obtained with the formula
d = proj(rho - gradient_step * gradient) - rho
then there are cases where<d , gradient> < 0, i.e.ddoes not follow the gradient anymore but goes backward. I suspect this is due to some conditions that are not fulfilled by the problem (convexity, continuity, condition on the projection, ...?), but I do not have any proof yet.
See https://sites.math.washington.edu/~burke/crs/408/notes/nlp/gpa.pdf
Fixing it would:
- improve performance of the
gradreconstruction method, - make the
gradreconstruction method more robust, as a fixed step of1e-5does not provide us any guarantee of convergence.