At linesearchmax, select alpha with minimum value#185
Merged
pkofod merged 3 commits intoJuliaNLSolvers:masterfrom Nov 19, 2025
Merged
At linesearchmax, select alpha with minimum value#185pkofod merged 3 commits intoJuliaNLSolvers:masterfrom
pkofod merged 3 commits intoJuliaNLSolvers:masterfrom
Conversation
This was referenced Nov 12, 2025
pkofod
approved these changes
Nov 17, 2025
pkofod
approved these changes
Nov 18, 2025
pkofod
reviewed
Nov 18, 2025
Co-authored-by: Patrick Kofod Mogensen <patrick.mogensen@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR modifies the behavior when Hager-Zhang line search hits the maximum number of iterations without being able to bracket the step size alpha. Prior to this PR, HZ returns the smallest alpha (
alphas[ia] == 0.0) in itsLineSearchException. After this PR, HZ returns the alpha associated with the lowest value, i.e.alphas[argmin(values)].This failure case can arise when using Optim with the
Manifoldfeature. Consider, for example, a sphere manifold. If the initial condition is the north pole and the optimizer wants to move to the equator, then an "infinite" step size alpha is formally required. This is because theretract!feature ofManifolduses simple projection rather than exponential transport. Since HZ line search cannot actually bracket an infinite alpha, it eventually gives up. When this happens, it's better to accept a very large step size rather than take a zero step size.Even after this PR is merged, it is still a bit tricky to make the Optim Manifold optimization reliable. If the line search fails, Optim's ConjugateGradient (or similar) will terminate with
termination_code(res) == TerminationCode.FailedLinesearch. Some user-side logic is needed to detect this case and restart CG.Another source of brittleness occurs when line search succeeds, but the step size alpha is divergently large (e.g., alpha of order 1e23 was showing up in some of my tests). In this case, CG's next line search is likely to fail, because its initial guess for alpha will be based on the previous line search's accepted alpha. Unfortunately, here, the HZ line search will not have enough iterations available to decrease alpha back to order 1. I found a good workaround for this case is to explicitly bound the maximum initial alpha by some constant
αmax. This is possible with, e.g.,For example, when optimizing on a sphere of unit radius, the choice
αmax=10.0seems fine.Finally, I'll mention something else that does not work. Rather than the bounded
alphaguessobject, I instead tried passinglinesearch=LineSearches.HagerZhang(; alphamax=10.0)to theConjugateGradientconstructor. This just leads to an unrecoverable assertion error, rather than aLineSearchException. So this is not a way to bound the maximum step size in the line search.