Calculate derivatives using Jacobian-vector products

[devmotion]

A sketch of how JuliaNLSolvers/NLSolversBase.jl#168 could be used in LineSearches.jl.

A possible downside could be that gradient(df) would not return the gradient of the objective function at x_new anymore, so would be computed in e.g. Optim.jl after the line search is completed.

[pkofod]

A possible downside could be that gradient(df) would not return the gradient of the objective function at x_new anymore, so would be computed in e.g. Optim.jl after the line search is completed.

yes that is a downside, however I believe that in many use cases the final iteration of the line search will not terminate with the evaluation of phidphi but rather this is only used if we need to expand the interval. If the wolfe conditions are already satisfied at 0 we will not need to do any additional gradient evaluations. If we do, it seems "likely" that you'll expand the interval and then backtrack / search towards 0. Especially as many default settings aggressively expand the interval to quickly bound a minimum.

I users do not provide a directional derivative / jvp function it will fallback to gradient!(d, x) and in that case it is cached. So users should only supply jvp if they think that jvp is so cheap compared to g(x)'v that if you happen to accept the first point where we calculate jvp you have to then calculate the gradient.

[pkofod]

So, for example, I don't think this should be changed https://github.com/JuliaNLSolvers/Optim.jl/blob/b367149cfec6e339ca3790714d1625d82a8b3f69/src/utilities/perform_linesearch.jl#L42 that relies og g(x) being readily available (free/lookup) because we already evaluated it in the past iteration / first iteration to check for convergence.

[pkofod]

We have benchmarks in Optim, but not here. Would be interesting to see the effect, but on the other hand maybe everything is too fast.

[pkofod]

This is ready for review I suppose?

[devmotion]

No, NLSolversBase is not ready yet.