Pass optimization state to callbacks

[devmotion]

In some downstream application I want to define a custom callback that requires information about the current state x (and ideally also f(x) but I could work around this by closing over the objective function and reevaluating it...) However, currently, that information is not provided to callbacks unless extended_trace = true:

Optim.jl/src/utilities/trace.jl

Lines 14 to 18 in 540c97b

	if options.extended_trace
	dt["x"] = copy(state.x)
	dt["g(x)"] = copy(state.g_x)
	dt["Current step size"] = state.alpha
	end

Since users may change the tracing options, this would mean that the callback would be broken when users change this option.

With #1212, the current x (and f(x) and if available its gradient and/or Hessian) are all included in the state of the optimizer (at least for multivariate optimization problems, univariate ones are a bit special it seems). Hence in this PR I propose to

• fully decouple callbacks from tracing
• forwarding the full state of the optimization algorithm to the callback

[github-actions]

Benchmark Results (Julia vlts)

Time benchmarks

	master	6818352...	master / 6818352...
multivariate/solvers/first_order/AdaMax	0.545 ± 0.0098 ms	0.545 ± 0.0096 ms	0.999 ± 0.025
multivariate/solvers/first_order/Adam	0.543 ± 0.0092 ms	0.543 ± 0.0099 ms	0.998 ± 0.025
multivariate/solvers/first_order/BFGS	0.264 ± 0.0086 ms	0.264 ± 0.0085 ms	1 ± 0.046
multivariate/solvers/first_order/ConjugateGradient	0.18 ± 0.0033 ms	0.181 ± 0.0032 ms	0.994 ± 0.025
multivariate/solvers/first_order/GradientDescent	1.55 ± 0.017 ms	1.56 ± 0.014 ms	0.993 ± 0.014
multivariate/solvers/first_order/LBFGS	0.231 ± 0.008 ms	0.233 ± 0.0081 ms	0.992 ± 0.049
multivariate/solvers/first_order/MomentumGradientDescent	2.15 ± 0.017 ms	2.18 ± 0.019 ms	0.988 ± 0.011
multivariate/solvers/first_order/NGMRES	0.426 ± 0.011 ms	0.428 ± 0.011 ms	0.995 ± 0.036
time_to_load	0.386 ± 0.0088 s	0.372 ± 0.0042 s	1.04 ± 0.026

Memory benchmarks

	master	6818352...	master / 6818352...
multivariate/solvers/first_order/AdaMax	0.34 k allocs: 7.19 kB	0.34 k allocs: 7.19 kB	1
multivariate/solvers/first_order/Adam	0.34 k allocs: 7.19 kB	0.34 k allocs: 7.19 kB	1
multivariate/solvers/first_order/BFGS	0.336 k allocs: 15 kB	0.336 k allocs: 15 kB	1
multivariate/solvers/first_order/ConjugateGradient	0.338 k allocs: 13.6 kB	0.338 k allocs: 13.6 kB	1
multivariate/solvers/first_order/GradientDescent	1.89 k allocs: 0.0713 MB	1.89 k allocs: 0.0713 MB	1
multivariate/solvers/first_order/LBFGS	0.317 k allocs: 14.2 kB	0.317 k allocs: 14.2 kB	1
multivariate/solvers/first_order/MomentumGradientDescent	2.24 k allocs: 0.077 MB	2.24 k allocs: 0.077 MB	1
multivariate/solvers/first_order/NGMRES	1.41 k allocs: 0.112 MB	1.41 k allocs: 0.112 MB	1
time_to_load	0.153 k allocs: 14.5 kB	0.153 k allocs: 14.5 kB	1

A plot of the benchmark results has been uploaded as an artifact at .

[codecov]

Codecov Report

❌ Patch coverage is 87.50000% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.79%. Comparing base (540c97b) to head (6818352).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
src/multivariate/solvers/constrained/samin.jl	60.00%	2 Missing ⚠️
...tivariate/solvers/constrained/ipnewton/interior.jl	88.88%	1 Missing ⚠️
src/univariate/solvers/brent.jl	87.50%	1 Missing ⚠️
src/univariate/solvers/golden_section.jl	87.50%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1224      +/-   ##
==========================================
- Coverage   86.83%   86.79%   -0.04%     
==========================================
  Files          44       44              
  Lines        3540     3552      +12     
==========================================
+ Hits         3074     3083       +9     
- Misses        466      469       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[pkofod]

The proposed strategy is exactly in line with the thinking I had on this topic. That tracing and callbacks should be decoupled, and that the full state should be available (with a big warning for users not to mess with the state unless we copy by default although that comes at a significant overhead).