Unify `Optimizer` and `Solver` interface

[benedict-96]

This is related to issues #60, #40 and (vaguely) #22.

At the moment the Optimizer and NewtonSolver structs are both subtyped from NonlinearSolver, share some properties but differ in others. I tried to visualize this with the following tikz pictures:

Same colors indicate similar/equivalent functionality.
As can be seen the differences are:

1. the Hessian and the Jacobian are not stored at the same level,
2. the solver contains a result object, but the solver doesn't,
3. the solver stores a linesearch, but the optimizer doesn't,
4. the optimizer stores an "algorithm" and the solver doesn't. The "algorithm" in the optimizer mainly concerns how the Hessian is computed/approximated. The equivalent in the solver case would probably be the "mode" keyword that we parse to the nonlinearsystem and that determines how the Jacobian is computed.

Some of these discrepancies between solvers and optimizers may make sense, but I think many do not. I think we should settle on a more or less consistent interface.

[benedict-96]

The Optimizer struct should probably also not be subtyped from NonlinearSolver.

[michakraus]

I guess this should be something like

abstract type AbstractSolver end
abstract type AbstractNonlinearSolver <: AbstractSolver end
abstract type NonlinearSolver <: AbstractNonlinearSolver end
abstract type NewtonSolver <: NonlinearSolver end
abstract type Optimizer <: AbstractNonlinearSolver end
abstract type LinearSolver <: AbstractSolver end

Would this make sense or does it seem a bit like overkill?

[benedict-96]

Two different propositions:

Proposition 1: Making Solvers and Optimizers distinct

I think it makes sense to have:

abstract type AbstractSolver end
abstract type NonlinearSolver <: AbstractSolver end
abstract type NewtonSolver <: NonlinearSolver end
abstract type LinearSolver <: AbstractSolver end
abstract type Optimizer end

And here we would make optimizers distinct from solvers. At the moment I understand solvers (at least in the SimpleSolvers context) to be something that solves a system of equations (be it linear or nonlinear) up to machine precision (i.e. they are root-finding algorithms). Optimizers on the other hand try to find a (most of the time local, but ideally global) minimum of a scalar function and our primary use for that are machine learning tasks, especially training neural networks.

Having said that the wikipedia definition of solvers encompasses optimizers, but also solvers for ODEs/PDEs, hence we could also use a different terminology.

Proposition 2: Subtyping RootFinders and Optimizers from AbstractSolvers

We could do:

abstract type AbstractSolver end
abstract type RootFinder <: AbstractSolver end
abstract type NonlinearRootFinder <: RootFinder end
abstract type LinearRootFinder <: RootFinder end
abstract type NewtonSolver <: NonlinearRootFinder end
abstract type Optimizer <: AbstractSolver end

[benedict-96]

Just for bookkeeping; we now settled on the following:

abstract type AbstractSolver end
abstract type LinearSolver <: AbstractSolver end
abstract type NonlinearSolver <: AbstractSolver end
abstract type NewtonSolver <: NonlinearSolver end
abstract type Optimizer <: AbstractSolver end

[benedict-96]

The only necessary change for the step above was the change made in commit f7ea1f9 (which is part of #44).

[benedict-96]

I just realized that in the mindmap above I forgot to include the LinesearchState that is part of the NewtonOptimizerState <: OptimizationAlgorithm. Note that an instance of LinesearchState is stored in solver.linesearch, but the equivalent for the optimizer would be optimizer.state.linesearch, which should probably be unified.

So I think we could do two things:

1. have a field linesearch in both NonlinearSolvers and Optimizers that stores a LinesearchState as well as a state that stores other information (optimizer/solver history etc.)
2. have a field linesearch in the OptimizerState and NonlinearSolverState.

This is also related to #60 and obliquely to #20.

[michakraus]

When me make these changes, we need to be careful to properly separate all persistent and non-persistent data once and for all. This include separating all solution dependent data structures from the solver/optimizer data structure, i.e., history and state variables should not be stored in a solver/optimizer but should be passed to its solve_step! methods.

In GeometricIntegrators, we use the SolutionStep data structure for this. It stores:

• the current solution (and vector fields),
• a history of solutions of a length determined by the integration method,
• parameters of the equation being solved,
• the internal state of the integrator if necessary for the computation of consecutive steps.

We should adopt a similar strategy here.

The internal state is stored as a NamedTuple. Each integrator can implement two methods internal_variables and copy_internal_variables which return a NamedTuple whose entries correspond to the internal state of an integrator and copy the state to an existing such NamedTuple, respectively. By default it is just empty.

[benedict-96]

1. optimization_step!(x, optimizer_state , optimizer)
2. Optimizer requires a state in general, Solver does not.
3. for each optimizer/solver Method we need a cache method and a state (potentially).

[benedict-96]

In eff3c45 I introduced OptimizerMethods. BFGS or DFP are OptimizerMethods for example. These then largely influence the way the Hessian is computed. In some cases, i.e. for the Adam optimizer, we do not have a Hessian however.
For the solvers the equivalent of the Hessian would probably be the Jacobian (as is also indicated in the tikz picture above). This is at the moment however stored in the NonlinearSystem which should probably not be that way.

[benedict-96]

After merging #66 we have:

However note that in☝️ the equivalent objects are called algorithm in the optimizer case and method in the solver case.