Document optimization state and callback functionality. Change initial_state to initial_x in the user-facing API and export initial_state.

Author: pkofod

docs/src/user/callbacks.md

@@ -0,0 +1,27 @@
+## Callbacks
+
+Callbacks are functions that are called at certain points during the optimization process. They can be used to monitor progress, log information, or implement custom stopping criteria. Callbacks are called each **iteration** of an algorithm. By iteration, we mean each time the algorithm updates its current estimate of the solution and checks for convergence. This structure is not necessarily uniquely defined for all algorithms. For example, we could in principle call the callback function within the line search algorithm, or for each sampled point in a derivative-free algorithm.
+
+### Callback Function Example
+
+We show a simple example of a callback function that prints the current iteration number and objective value at each iteration.
+
+```julia
+using Optim
+function my_callback(state)
+    print(" Objective Value: ", state.f_x)
+    println(" at state x: ", state.x)
+    return false  # Return true to stop the optimization
+end
+function objective(x)
+    return (x[1]-2)^2 + (x[2]-3)^2
+end
+
+initial_x = [0.0, 0.0]
+method = BFGS()
+options = Optim.Options(callback=my_callback)
+d = OnceDifferentiable(objective, initial_x)
+
+optstate = initial_state(method, options, d, initial_x)
+result = optimize(d, initial_x, method, options, optstate)
+```

docs/src/user/minimization.md

@@ -213,7 +213,7 @@ Defined for multivariate optimization:
 * `x_converged(res)`
 * `f_converged(res)`
 * `g_converged(res)`
-* `initial_state(res)`
+* `initial_x(res)`
 
 Defined for `NelderMead` with the option `trace_simplex=true`:
 

docs/src/user/optstate.md

@@ -0,0 +1,43 @@
+## Optimization State
+
+Each algorithm in Optim.jl maintains an optimization state that encapsulates all relevant information about the current iteration of the optimization process. This state is represented by the sub-types of `Optim.OptimizationState` and contains various fields that provide insights into the progress of the optimization and any information needed to maintain and update the search direction.
+
+### Exceptions
+
+Currently, there are two main exceptions to this structure:
+- **SAMIN**: This algorithm is currently not written using the main `optimize` loop and does not maintain an `OptimizationState`.
+- **Univariate Optimization Algorithms**: These algorithms do not use the `OptimizationState` structure as they also do not use the main `optimize` loop.
+
+The exceptions matter mostly for users who want to pre-allocate the `OptimizationState` for performance reasons. In these cases, users should check the documentation of the specific algorithm they are using to see if it supports pre-allocation. It also matters for users who want to make use of the callback functionality, as the callback functions receive the `OptimizationState` as an argument. If the algorithm does not use the `OptimizationState`, the callback will instead receive a `NamedTuple` with relevant information and the callback functions should not use type annotations for their arguments based on the `OptimizationState` hierarchy.
+
+### Using the Optimization State
+
+As mentioned above, the optimization state is passed to callback functions during the optimization process. Users can access various fields of the state to monitor progress or implement custom logic based on the current state of the optimization. It is also possible to pre-allocate the optimization state if users which to re-use it across multiple optimization runs for performance reasons. This can be done using the `initial_state` function, which takes the optimization method, options, differentiable object, and initial parameters as arguments.
+
+#### Initial State Example
+```julia
+using Optim
+function objective(x)
+    return (x[1]-2)^2 + (x[2]-3)^2
+end
+
+initial_x = [0.0, 0.0]
+method = BFGS()
+options = Optim.Options(callback=my_callback)
+d = OnceDifferentiable(objective, initial_x)
+
+# Pre-allocate the optimization state
+optstate = initial_state(method, options, d, initial_x)
+
+# Verify that the state has the properties f_x and x
+hasproperty(optstate, :f_x)  # true
+hasproperty(optstate, :x)    # true
+
+result = optimize(d, initial_x, method, options, optstate)
+```
+
+After the optimization is complete, the state has been updated as part of the optimization process and contains information about the final iteration. Users can access fields of the state to retrieve information about the final state. For example, we can verify that the final objective value matches the value stored in the state.
+
+```julia
+@assert optstate.f_x == Optim.minimum(result)
+```

src/Optim.jl

@@ -85,9 +85,9 @@ export optimize,
     # Re-export constraint types from NLSolversBase
     TwiceDifferentiableConstraints,
 
-    # I don't think these should be here [pkofod]
     OptimizationState,
     OptimizationTrace,
+    initial_state,
 
     # Optimization algorithms
     ## Zeroth order methods (heuristics)

src/api.jl

@@ -159,9 +159,9 @@ g_abstol(r::MultivariateOptimizationResults) = r.g_abstol
 g_residual(r::MultivariateOptimizationResults) = r.g_residual
 
 
-initial_state(r::OptimizationResults) =
-    error("initial_state is not implemented for $(summary(r)).")
-initial_state(r::MultivariateOptimizationResults) = r.initial_x
+initial_x(r::OptimizationResults) =
+    error("initial_x is not implemented for $(summary(r)).")
+initial_x(r::MultivariateOptimizationResults) = r.initial_x
 
 lower_bound(r::OptimizationResults) =
     error("lower_bound is not implemented for $(summary(r)).")

src/maximize.jl

@@ -89,7 +89,7 @@ for api_method in (
     :rel_tol,
     :abs_tol,
     :iterations,
-    :initial_state,
+    :initial_x,
     :converged,
     :x_tol,
     :x_abstol,

src/multivariate/optimize/interface.jl

@@ -39,7 +39,7 @@ fallback_method(d::OnceDifferentiable) = LBFGS()
 fallback_method(d::TwiceDifferentiable) = Newton()
 
 # promote the objective (tuple of callables or an AbstractObjective) according to method requirement
-promote_objtype(method, initial_x, autodiff::ADTypes.AbstractADType, inplace::Bool, args...) =
+promote_objtype(method, x0, autodiff::ADTypes.AbstractADType, inplace::Bool, args...) =
     error("No default objective type for $method and $args.")
 # actual promotions, notice that (args...) captures FirstOrderOptimizer and NonDifferentiable, etc
 promote_objtype(method::ZerothOrderOptimizer, x, autodiff::ADTypes.AbstractADType, inplace::Bool, args...) =
@@ -138,156 +138,156 @@ promote_objtype(
 # if no method or options are present
 function optimize(
     f,
-    initial_x::AbstractArray;
+    x0::AbstractArray;
     inplace::Bool = true,
     autodiff::ADTypes.AbstractADType = DEFAULT_AD_TYPE,
 )
     method = fallback_method(f)
-    d = promote_objtype(method, initial_x, autodiff, inplace, f)
+    d = promote_objtype(method, x0, autodiff, inplace, f)
 
     options = Options(; default_options(method)...)
-    optimize(d, initial_x, method, options)
+    optimize(d, x0, method, options)
 end
 function optimize(
     f,
     g,
-    initial_x::AbstractArray;
+    x0::AbstractArray;
     autodiff::ADTypes.AbstractADType = DEFAULT_AD_TYPE,
     inplace::Bool = true,
 )
 
     method = fallback_method(f, g)
 
-    d = promote_objtype(method, initial_x, autodiff, inplace, f, g)
+    d = promote_objtype(method, x0, autodiff, inplace, f, g)
 
     options = Options(; default_options(method)...)
-    optimize(d, initial_x, method, options)
+    optimize(d, x0, method, options)
 end
 function optimize(
     f,
     g,
     h,
-    initial_x::AbstractArray;
+    x0::AbstractArray;
     inplace::Bool = true,
     autodiff::ADTypes.AbstractADType = DEFAULT_AD_TYPE, 
 )
     method = fallback_method(f, g, h)
-    d = promote_objtype(method, initial_x, autodiff, inplace, f, g, h)
+    d = promote_objtype(method, x0, autodiff, inplace, f, g, h)
 
     options = Options(; default_options(method)...)
-    optimize(d, initial_x, method, options)
+    optimize(d, x0, method, options)
 end
 
 # no method supplied with objective
 function optimize(
     d::T,
-    initial_x::AbstractArray,
+    x0::AbstractArray,
     options::Options,
 ) where {T<:AbstractObjective}
-    optimize(d, initial_x, fallback_method(d), options)
+    optimize(d, x0, fallback_method(d), options)
 end
 # no method supplied with inplace and autodiff keywords becauase objective is not supplied
 function optimize(
     f,
-    initial_x::AbstractArray,
+    x0::AbstractArray,
     options::Options;
     inplace::Bool = true,
     autodiff::ADTypes.AbstractADType = DEFAULT_AD_TYPE,
 )
     method = fallback_method(f)
-    d = promote_objtype(method, initial_x, autodiff, inplace, f)
-    optimize(d, initial_x, method, options)
+    d = promote_objtype(method, x0, autodiff, inplace, f)
+    optimize(d, x0, method, options)
 end
 function optimize(
     f,
     g,
-    initial_x::AbstractArray,
+    x0::AbstractArray,
     options::Options;
     inplace::Bool = true,
     autodiff::ADTypes.AbstractADType = DEFAULT_AD_TYPE,
 )
 
     method = fallback_method(f, g)
-    d = promote_objtype(method, initial_x, autodiff, inplace, f, g)
-    optimize(d, initial_x, method, options)
+    d = promote_objtype(method, x0, autodiff, inplace, f, g)
+    optimize(d, x0, method, options)
 end
 function optimize(
     f,
     g,
     h,
-    initial_x::AbstractArray{T},
+    x0::AbstractArray{T},
     options::Options;
     inplace::Bool = true,
     autodiff::ADTypes.AbstractADType = DEFAULT_AD_TYPE,
 ) where {T}
     method = fallback_method(f, g, h)
-    d = promote_objtype(method, initial_x, autodiff, inplace, f, g, h)
+    d = promote_objtype(method, x0, autodiff, inplace, f, g, h)
 
-    optimize(d, initial_x, method, options)
+    optimize(d, x0, method, options)
 end
 
 # potentially everything is supplied (besides caches)
 function optimize(
     f,
-    initial_x::AbstractArray,
+    x0::AbstractArray,
     method::AbstractOptimizer,
     options::Options = Options(; default_options(method)...);
     inplace::Bool = true,
     autodiff::ADTypes.AbstractADType = DEFAULT_AD_TYPE,
 )
-    d = promote_objtype(method, initial_x, autodiff, inplace, f)
-    optimize(d, initial_x, method, options)
+    d = promote_objtype(method, x0, autodiff, inplace, f)
+    optimize(d, x0, method, options)
 end
 function optimize(
     f,
     c::AbstractConstraints,
-    initial_x::AbstractArray,
+    x0::AbstractArray,
     method::AbstractOptimizer,
     options::Options = Options(; default_options(method)...);
     inplace::Bool = true,
     autodiff::ADTypes.AbstractADType = DEFAULT_AD_TYPE,
 )
 
-    d = promote_objtype(method, initial_x, autodiff, inplace, f)
-    optimize(d, c, initial_x, method, options)
+    d = promote_objtype(method, x0, autodiff, inplace, f)
+    optimize(d, c, x0, method, options)
 end
 function optimize(
     f,
     g,
-    initial_x::AbstractArray,
+    x0::AbstractArray,
     method::AbstractOptimizer,
     options::Options = Options(; default_options(method)...);
     inplace::Bool = true,
     autodiff::ADTypes.AbstractADType = DEFAULT_AD_TYPE,
 )
-    d = promote_objtype(method, initial_x, autodiff, inplace, f, g)
+    d = promote_objtype(method, x0, autodiff, inplace, f, g)
 
-    optimize(d, initial_x, method, options)
+    optimize(d, x0, method, options)
 end
 function optimize(
     f,
     g,
     h,
-    initial_x::AbstractArray,
+    x0::AbstractArray,
     method::AbstractOptimizer,
     options::Options = Options(; default_options(method)...);
     inplace::Bool = true,
     autodiff::ADTypes.AbstractADType = DEFAULT_AD_TYPE,
 
 )
-    d = promote_objtype(method, initial_x, autodiff, inplace, f, g, h)
+    d = promote_objtype(method, x0, autodiff, inplace, f, g, h)
 
-    optimize(d, initial_x, method, options)
+    optimize(d, x0, method, options)
 end
 
 function optimize(
     d::D,
-    initial_x::AbstractArray,
+    x0::AbstractArray,
     method::SecondOrderOptimizer,
     options::Options = Options(; default_options(method)...);
     inplace::Bool = true,
     autodiff::ADTypes.AbstractADType = DEFAULT_AD_TYPE,
 ) where {D<:Union{NonDifferentiable,OnceDifferentiable}}
-    d = promote_objtype(method, initial_x, autodiff, inplace, d)
-    optimize(d, initial_x, method, options)
+    d = promote_objtype(method, x0, autodiff, inplace, d)
+    optimize(d, x0, method, options)
 end