Updated sciml_train information. (#500)

* Updated sciml_train information.

* Replaced sciml_train with GalacticOptim

Author: mkg33

docs/Project.toml

@@ -2,4 +2,4 @@
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
 
 [compat]
-Documenter = "0.24.2"
+Documenter = "0.26.1"

docs/make.jl

@@ -74,7 +74,7 @@ makedocs(
             "FastChain" => "FastChain.md",
             "Smoothed Collocation" => "Collocation.md",
             "GPUs" => "GPUs.md",
-            "sciml_train" => "Scimltrain.md",
+            "GalacticOptim.jl" => "GalacticOptim.md",
 
         ],
         "Benchmarks" => "Benchmark.md"

docs/src/GalacticOptim.md

@@ -0,0 +1,129 @@
+# GalacticOptim.jl
+
+*Important:* Please note that `sciml_train` has now been replaced by
+[GalacticOptim.jl](https://github.com/SciML/GalacticOptim.jl). The conversion to
+GalacitcOptim.jl-style should not pose too many problems! Consult
+[this](https://diffeqflux.sciml.ai/dev/examples/neural_ode_sciml/) tutorial
+to see how an optimization problem can be set up and/or also read the updated
+[GalacticOptim.jl documentation](https://galacticoptim.sciml.ai/stable/)
+to explore more options in detail.
+
+GalacticOptim.jl is a package with a scope that is beyond your normal global optimization
+package. GalacticOptim.jl seeks to bring together all of the optimization packages
+it can find, local and global, into one unified Julia interface. This means, you
+learn one package and you learn them all! GalacticOptim.jl adds a few high-level
+features, such as integrating with automatic differentiation, to make its usage
+fairly simple for most cases, while allowing all of the options in a single
+unified interface.
+
+##### Note: This package is still in active development.
+
+## Installation
+
+Assuming that you already have Julia correctly installed, it suffices to import
+GalacticOptim.jl in the standard way:
+
+```julia
+import Pkg; Pkg.add("GalacticOptim")
+```
+The packages relevant to the core functionality of GalacticOptim.jl will be imported
+accordingly and, in most cases, you do not have to worry about the manual
+installation of dependencies. Below is the list of packages that need to be
+installed explicitly if you intend to use the specific optimization algorithms
+offered by them:
+
+- [BlackBoxOptim.jl](https://github.com/robertfeldt/BlackBoxOptim.jl) (solver: `BBO()`)
+- [NLopt.jl](https://github.com/JuliaOpt/NLopt.jl) (usage via the NLopt API;
+see also the available [algorithms](https://nlopt.readthedocs.io/en/latest/NLopt_Algorithms/))
+- [MultistartOptimization.jl](https://github.com/tpapp/MultistartOptimization.jl)
+(see also [this documentation](https://juliahub.com/docs/MultistartOptimization/cVZvi/0.1.0/))
+- [QuadDIRECT.jl](https://github.com/timholy/QuadDIRECT.jl)
+- [Evolutionary.jl](https://github.com/wildart/Evolutionary.jl) (see also [this documentation](https://wildart.github.io/Evolutionary.jl/dev/))
+- [CMAEvolutionStrategy.jl](https://github.com/jbrea/CMAEvolutionStrategy.jl)
+
+## Tutorials and Documentation
+
+For information on using the package,
+[see the stable documentation](https://galacticoptim.sciml.ai/stable/). Use the
+[in-development documentation](https://galacticoptim.sciml.ai/dev/) for the version of
+the documentation, which contains the unreleased features.
+
+## Examples
+
+```julia
+ using GalacticOptim, Optim
+ rosenbrock(x,p) =  (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
+ x0 = zeros(2)
+ p  = [1.0,100.0]
+
+ prob = OptimizationProblem(rosenbrock,x0,p)
+ sol = solve(prob,NelderMead())
+
+
+ using BlackBoxOptim
+ prob = OptimizationProblem(rosenbrock, x0, p, lb = [-1.0,-1.0], ub = [1.0,1.0])
+ sol = solve(prob,BBO())
+```
+
+Note that Optim.jl is a core dependency of GalaticOptim.jl. However, BlackBoxOptim.jl
+is not and must already be installed (see the list above).
+
+The output of the first optimization task (with the `NelderMead()` algorithm)
+is given below:
+
+```julia
+* Status: success
+
+* Candidate solution
+   Final objective value:     3.525527e-09
+
+* Found with
+   Algorithm:     Nelder-Mead
+
+* Convergence measures
+   √(Σ(yᵢ-ȳ)²)/n ≤ 1.0e-08
+
+* Work counters
+   Seconds run:   0  (vs limit Inf)
+   Iterations:    60
+   f(x) calls:    118
+```
+We can also explore other methods in a similar way:
+
+```julia
+ f = OptimizationFunction(rosenbrock, GalacticOptim.AutoForwardDiff())
+ prob = OptimizationProblem(f, x0, p)
+ sol = solve(prob,BFGS())
+```
+For instance, the above optimization task produces the following output:
+
+```julia
+* Status: success
+
+* Candidate solution
+   Final objective value:     7.645684e-21
+
+* Found with
+   Algorithm:     BFGS
+
+* Convergence measures
+   |x - x'|               = 3.48e-07 ≰ 0.0e+00
+   |x - x'|/|x'|          = 3.48e-07 ≰ 0.0e+00
+   |f(x) - f(x')|         = 6.91e-14 ≰ 0.0e+00
+   |f(x) - f(x')|/|f(x')| = 9.03e+06 ≰ 0.0e+00
+   |g(x)|                 = 2.32e-09 ≤ 1.0e-08
+
+* Work counters
+   Seconds run:   0  (vs limit Inf)
+   Iterations:    16
+   f(x) calls:    53
+   ∇f(x) calls:   53
+```
+
+```julia
+ prob = OptimizationProblem(f, x0, p, lb = [-1.0,-1.0], ub = [1.0,1.0])
+ sol = solve(prob, Fminbox(GradientDescent()))
+```
+The examples clearly demonstrate that GalacticOptim.jl provides an intuitive
+way of specifying optimization tasks and offers a relatively
+easy access to a wide range of optimization algorithms.

docs/src/Scimltrain.md

@@ -1,17 +1,17 @@
-# Neural Ordinary Differential Equations with sciml_train
+# Neural Ordinary Differential Equations with GalacticOptim.jl
 
 We can use DiffEqFlux.jl to define, solve, and train neural ordinary
 differential equations. A neural ODE is an ODE where a neural network defines
 its derivative function. Thus for example, with the multilayer perceptron neural
-network `Chain(Dense(2, 50, tanh), Dense(50, 2))`,
+network `Chain(Dense(2, 50, tanh), Dense(50, 2))`, we obtain the following results.
 
 ## Copy-Pasteable Code
 
-Before getting to the explanation, here's some code to start with. We will follow
-wil a full explanation of the definition and training process:
+Before getting to the explanation, here's some code to start with. We will
+follow a full explanation of the definition and training process:
 
 ```julia
-using DiffEqFlux, OrdinaryDiffEq, Flux, Optim, Plots
+using DiffEqFlux, OrdinaryDiffEq, Flux, Optim, Plots, GalacticOptim
 
 u0 = Float32[2.0; 0.0]
 datasize = 30
@@ -52,15 +52,24 @@ callback = function (p, l, pred; doplot = true)
   return false
 end
 
-result_neuralode = DiffEqFlux.sciml_train(loss_neuralode, prob_neuralode.p,
-                                          ADAM(0.05), cb = callback,
-                                          maxiters = 300)
+# use GalacticOptim.jl to solve the problem
+adtype = GalacticOptim.AutoZygote()
+
+optf = GalacticOptim.OptimizationFunction((x, p) -> loss_neuralode(x), adtype)
+optfunc = GalacticOptim.instantiate_function(optf, prob_neuralode.p, adtype, nothing)
+optprob = GalacticOptim.OptimizationProblem(optfunc, prob_neuralode.p)
+
+result_neuralode = GalacticOptim.solve(optprob,
+                                       ADAM(0.05),
+                                       cb = callback,
+                                       maxiters = 300)
+
+optprob2 = remake(optprob,u0 = result_neuralode.minimizer)
 
-result_neuralode2 = DiffEqFlux.sciml_train(loss_neuralode,
-                                           result_neuralode.minimizer,
-                                           LBFGS(),
-                                           cb = callback,
-                                           allow_f_increases = false)
+result_neuralode2 = GalacticOptim.solve(optprob2,
+                                        LBFGS(),
+                                        cb = callback,
+                                        allow_f_increases = false)
 ```
 
 ![Neural ODE](https://user-images.githubusercontent.com/1814174/88589293-e8207f80-d026-11ea-86e2-8a3feb8252ca.gif)
@@ -147,15 +156,25 @@ We then train the neural network to learn the ODE.
 Here we showcase starting the optimization with `ADAM` to more quickly find a
 minimum, and then honing in on the minimum by using `LBFGS`. By using the two
 together, we are able to fit the neural ODE in 9 seconds! (Note, the timing
-commented out the plotting).
+commented out the plotting). You can easily incorporate the procedure below to
+set up custom optimization problems. For more information on the usage of
+[GalacticOptim.jl](https://github.com/SciML/GalacticOptim.jl), please consult
+[this](https://galacticoptim.sciml.ai/stable/) documentation.
 
 ```julia
 # Train using the ADAM optimizer
-result_neuralode = DiffEqFlux.sciml_train(loss_neuralode, prob_neuralode.p,
-                                          ADAM(0.05), cb = callback,
-                                          maxiters = 300)
+adtype = GalacticOptim.AutoZygote()
 
-* Status: failure (reached maximum number of iterations)
+optf = GalacticOptim.OptimizationFunction((x, p) -> loss_neuralode(x), adtype)
+optfunc = GalacticOptim.instantiate_function(optf, prob_neuralode.p, adtype, nothing)
+optprob = GalacticOptim.OptimizationProblem(optfunc, prob_neuralode.p)
+
+result_neuralode = GalacticOptim.solve(optprob,
+                                       ADAM(0.05),
+                                       cb = callback,
+                                       maxiters = 300)
+# output
+* Status: success
 
 * Candidate solution
    Minimizer: [4.38e-01, -6.02e-01, 4.98e-01,  ...]
@@ -164,19 +183,6 @@ result_neuralode = DiffEqFlux.sciml_train(loss_neuralode, prob_neuralode.p,
 * Found with
    Algorithm:     ADAM
    Initial Point: [-3.02e-02, -5.40e-02, 2.78e-01,  ...]
-
-* Convergence measures
-   |x - x'|               = NaN ≰ 0.0e+00
-   |x - x'|/|x'|          = NaN ≰ 0.0e+00
-   |f(x) - f(x')|         = NaN ≰ 0.0e+00
-   |f(x) - f(x')|/|f(x')| = NaN ≰ 0.0e+00
-   |g(x)|                 = NaN ≰ 0.0e+00
-
-* Work counters
-   Seconds run:   5  (vs limit Inf)
-   Iterations:    300
-   f(x) calls:    300
-   ∇f(x) calls:   300
 ```
 
 We then complete the training using a different optimizer starting from where
@@ -185,12 +191,13 @@ halt when near the minimum.
 
 ```julia
 # Retrain using the LBFGS optimizer
-result_neuralode2 = DiffEqFlux.sciml_train(loss_neuralode,
-                                           result_neuralode.minimizer,
-                                           LBFGS(),
-                                           cb = callback,
-                                           allow_f_increases = false)
+optprob2 = remake(optprob,u0 = result_neuralode.minimizer)
 
+result_neuralode2 = GalacticOptim.solve(optprob2,
+                                        LBFGS(),
+                                        cb = callback,
+                                        allow_f_increases = false)
+# output
 * Status: success
 
 * Candidate solution
@@ -239,10 +246,10 @@ my_neural_ode_prob = solve(prob, Tsit5(), args...; kwargs...)
 
 and then one would use `solve` for the prediction like in other tutorials.
 
-In total, the from scratch form looks like:
+In total, the 'from-scratch' form looks like:
 
 ```julia
-using DiffEqFlux, OrdinaryDiffEq, Flux, Optim, Plots
+using DiffEqFlux, OrdinaryDiffEq, Flux, Optim, Plots, GalacticOptim
 
 u0 = Float32[2.0; 0.0]
 datasize = 30
@@ -260,13 +267,10 @@ ode_data = Array(solve(prob_trueode, Tsit5(), saveat = tsteps))
 dudt2 = FastChain((x, p) -> x.^3,
                   FastDense(2, 50, tanh),
                   FastDense(50, 2))
-neural_ode_f(u,p,t) = dudt2(u,p)
-pinit = initial_params(dudt2)
-prob = ODEProblem(neural_ode_f, u0, tspan, pinit)
+prob_neuralode = NeuralODE(dudt2, tspan, Tsit5(), saveat = tsteps)
 
 function predict_neuralode(p)
-  tmp_prob = remake(prob,p=p)
-  Array(solve(tmp_prob,Tsit5(),saveat=tsteps))
+  Array(prob_neuralode(u0, p))
 end
 
 function loss_neuralode(p)
@@ -286,13 +290,21 @@ callback = function (p, l, pred; doplot = true)
   return false
 end
 
-result_neuralode = DiffEqFlux.sciml_train(loss_neuralode, pinit,
-                                          ADAM(0.05), cb = callback,
-                                          maxiters = 300)
+adtype = GalacticOptim.AutoZygote()
+
+optf = GalacticOptim.OptimizationFunction((x, p) -> loss_neuralode(x), adtype)
+optfunc = GalacticOptim.instantiate_function(optf, prob_neuralode.p, adtype, nothing)
+optprob = GalacticOptim.OptimizationProblem(optfunc, prob_neuralode.p)
+
+result_neuralode = GalacticOptim.solve(optprob,
+                                       ADAM(0.05),
+                                       cb = callback,
+                                       maxiters = 300)
+
+optprob2 = remake(optprob,u0 = result_neuralode.minimizer)
 
-result_neuralode2 = DiffEqFlux.sciml_train(loss_neuralode,
-                                           result_neuralode.minimizer,
-                                           LBFGS(),
-                                           cb = callback,
-                                           allow_f_increases = false)
+result_neuralode2 = GalacticOptim.solve(optprob2,
+                                        LBFGS(),
+                                        cb = callback,
+                                        allow_f_increases = false)
 ```