Add iterator interface #745

tkf · 2019-09-09T21:44:29Z

This PR adds an iterator interface for optimizer. I tentatively call it Optim.optimizing. It provides an iterator-based API such that

let istate
    for istate′ in optimizing(args...; kwargs...)
        istate = istate′
    end
    OptimizationResults(istate)
end

is equivalent to optimize(args...; kwargs...) for multivariate optimizations.

close #599

tkf · 2019-09-09T21:47:35Z

src/multivariate/optimize/optimize.jl

+@with_kw struct IteratorState{IT <: OptimIterator, TR <: OptimizationTrace}
+    # Put `OptimIterator` in iterator state so that `OptimizationResults` can
+    # be constructed from `IteratorState`.
+    iter::IT


We can drop iter field from IteratorState if we change the API to:

let istate iter = optimizing(args...; kwargs...) for istate′ in iter istate = istate′ end OptimizationResults(iter, istate) # need to pass `iter` here end

Accessor functions like iteration_limit_reached(istate) and (f|g|h)_calls(istate) need .iter, too.

pkofod · 2019-09-09T22:00:06Z

Interesting, thanks. That took surprisingly few changes, but it was mostly written as an iterator already, just wasn't an actual iterator :laugh . Will have to have a more thorough look!

codecov · 2019-09-09T22:00:15Z

Codecov Report

Merging #745 into master will decrease coverage by 0.04%.
The diff coverage is 80.55%.

@@            Coverage Diff             @@
##           master     #745      +/-   ##
==========================================
- Coverage   81.46%   81.41%   -0.05%     
==========================================
  Files          43       43              
  Lines        2422     2438      +16     
==========================================
+ Hits         1973     1985      +12     
- Misses        449      453       +4

Impacted Files	Coverage Δ
src/multivariate/optimize/optimize.jl	`77.96% <80%> (-1.11%)`	⬇️
src/multivariate/optimize/interface.jl	`76.81% <81.81%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b51e742...cc594a8. Read the comment docs.

codecov · 2019-09-09T22:00:15Z

Codecov Report

❌ Patch coverage is 87.93103% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.01%. Comparing base (07faeee) to head (7620e4f).

Files with missing lines	Patch %	Lines
src/multivariate/optimize/optimize.jl	92.30%	5 Missing ⚠️
src/api.jl	71.42%	4 Missing ⚠️
src/multivariate/optimize/interface.jl	86.20%	4 Missing ⚠️
...c/multivariate/solvers/zeroth_order/nelder_mead.jl	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #745      +/-   ##
==========================================
- Coverage   85.73%   85.01%   -0.73%     
==========================================
  Files          46       45       -1     
  Lines        3604     3630      +26     
==========================================
- Hits         3090     3086       -4     
- Misses        514      544      +30

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tkf · 2019-09-09T23:31:44Z

src/multivariate/optimize/optimize.jl

+    @unpack_IteratorState istate
+    @unpack d, initial_x, method, options, state = iter

    after_while!(d, state, method, options)


Executing a mutating-function after_while! inside non-bang function OptimizationResults is not super great. But it looks like after_while! is a no-op mostly except for NelderMead so maybe it's OK? From a quick look, after_while! for NelderMead seems to be idempotent (so that, e.g., calling OptimizationResults(istate) inside a loop multiple times is OK). If that's the case, OptimizationResults(istate) practically has no side-effect?

But using a function like result! sounds good to me as well.

I got rid of after_while! so no more side effects

tkf · 2019-09-10T00:23:42Z

src/api.jl

-    if !isa(r.method, NelderMead)
-        throw(ArgumentError("There is no centroid involved in optimization using $(r.method). Please use x_trace(...) to grab the points from the trace."))
+function centroid_trace(r::Union{MultivariateOptimizationResults, IteratorState})
+    tr = trace(r)


I suppose tr = trace(r) should be added here? I think it'll throw UndefVarError otherwise. There are two more places I did this change.

I'm including these changes (adding tr = trace(r)) although they are not directly related to PR.

tkf · 2019-09-10T00:41:59Z

@pkofod After posted the PR, I realized that the first version didn't provide the way to access iteration state (e.g., getting "best minimizer so far") so I added accessor functions for iterator state by reusing the function names/code already defined for the result type. So the diff may not be as small as you remember now.

Anyway, I think I've implemented the APIs I need. I'm looking forward to hear your thoughts on this.

tkf · 2019-09-10T02:51:49Z

Aside: Using this API a few times, I realized that istate = istate′ pattern for leaking the iteration variable to the outer scope is rather ugly although I guess there is not much we can do. For a subset of problem, defining foreach like below is kind of useful.

function Base.foreach(f, iter::OptimIterator)
    local istate
    for istate′ in iter
        istate = istate′
        f(istate)
    end
    return istate
end

We can then use it as

result = foreach(optimizing(args...; kwargs...)) do istate
    @info "Minimizing..." minimizer(istate) minimum(istate)
end |> Optim.OptimizationResults

although this does not let us use break.

(Or we can implement transducer-compatible foldl so that foreach is as powerful as for loop 😉)

tkf · 2019-09-27T07:45:23Z

Friendly ping :)

Is there anything I can do to make it easier to merge?

pkofod · 2019-09-27T12:51:58Z

Sorry, I've had too much on my plate. I'll take a look soon.

tkf · 2019-09-27T22:47:34Z

Thanks!

cmey · 2023-03-14T21:20:02Z

🛎️ Can we have this?

I am still interested in plotting the evolution of the optimization : https://discourse.julialang.org/t/plotting-the-evolution-of-the-optimization-using-optim-jl/41978

src/multivariate/optimize/optimize.jl

github-actions · 2025-06-14T15:33:07Z

Benchmark Results

	master	`dc03af8`...	master / `dc03af8`...
multivariate/solvers/first_order/AdaMax	0.645 ± 0.0078 ms	0.654 ± 0.0081 ms	0.986 ± 0.017
multivariate/solvers/first_order/Adam	0.644 ± 0.008 ms	0.653 ± 0.0074 ms	0.987 ± 0.017
multivariate/solvers/first_order/BFGS	0.223 ± 0.0035 ms	0.226 ± 0.0044 ms	0.986 ± 0.025
multivariate/solvers/first_order/ConjugateGradient	0.048 ± 0.00062 ms	0.0491 ± 0.00069 ms	0.978 ± 0.019
multivariate/solvers/first_order/GradientDescent	1.7 ± 0.014 ms	1.74 ± 0.0063 ms	0.981 ± 0.0086
multivariate/solvers/first_order/LBFGS	0.219 ± 0.0036 ms	0.222 ± 0.0038 ms	0.988 ± 0.023
multivariate/solvers/first_order/MomentumGradientDescent	2.51 ± 0.018 ms	2.52 ± 0.011 ms	0.994 ± 0.0085
multivariate/solvers/first_order/NGMRES	0.553 ± 0.011 ms	0.569 ± 0.012 ms	0.973 ± 0.027
time_to_load	0.529 ± 0.0059 s	0.518 ± 0.0048 s	1.02 ± 0.015

Benchmark Plots

A plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR.
Go to "Actions"->"Benchmark a pull request"->[the most recent run]->"Artifacts" (at the bottom).

…test to succeed locally

…se the method is not in the state

devmotion · 2025-10-31T20:14:54Z

src/api.jl

-Base.summary(r::OptimizationResults) = summary(r.method) # might want to do more here than just return summary of the method used
+_method(r::OptimizationResults) = r.method
+
+Base.summary(r::Union{OptimizationResults, OptimIterator}) =


Base.summary should be overloaded as Base.summary(::IO, x)

devmotion · 2025-10-31T20:20:22Z

src/multivariate/optimize/interface.jl

+    # We can safely assume that `istate` is defined at this point.  That is to say,
+    # `OptimIterator` guarantees that `iterate(::OptimIterator) !== nothing`.


I think JET won't agree with this comment 😄

Generally, the code above seems a bit unfortunate... Maybe optimizing should return the iterator AND the initial state?

I also wonder, is there no utility in Julia for directly obtaining the last state of an iterator?

devmotion · 2025-10-31T20:22:16Z

src/multivariate/optimize/optimize.jl

+
+function Base.iterate(iter::OptimIterator, istate = nothing)
+    (; d, initial_x, method, options, state) = iter
+    if istate === nothing


IMO it would be cleaner to move this code to a separate Base.iterate(iter::OptimIterator) definition, I'm not a fan of "dummy" states such as nothing for the initial iteration as this is not part of the Base.iterate interface.

devmotion · 2025-10-31T20:24:46Z

src/multivariate/optimize/optimize.jl

+
+Base.IteratorSize(::Type{<:OptimIterator}) = Base.SizeUnknown()
+Base.IteratorEltype(::Type{<:OptimIterator}) = Base.HasEltype()
+Base.eltype(::Type{<:OptimIterator}) = IteratorState


Is it a problem that the element type is non-concrete here? Could it be defined in a way that the eltype is concrete?

Seems below TR is initialized as OptimizationTrace{typeof(value(d)),M}.

Add iterator interface

b10f27e

tkf commented Sep 9, 2019

View reviewed changes

Don't use at-eval

cc594a8

tkf added 2 commits September 9, 2019 15:54

Simplify OptimizationResults(istate)

d6a6d75

Add accessor functions

2117364

tkf force-pushed the iterator branch from 81b506e to 2117364 Compare September 9, 2019 23:31

tkf commented Sep 9, 2019

View reviewed changes

Directly define accessor functions

b7b63e8

tkf commented Sep 10, 2019

View reviewed changes

tkf mentioned this pull request Sep 10, 2019

Use Optim's iterator interface tkf/SteadyStateFit.jl#4

Open

3 tasks

Merge branch 'master' into iterator

8a64efb

Merge branch 'master' into iterator

0826be0

pkofod reviewed Jun 14, 2025

View reviewed changes