Parallel ParticleSwarm

[sgaure]

#1124

This PR enhances ParticleSwarm with a new keyword parallel::Bool = false. If set to true, the objective function will be called with a vector to fill in, and a matrix of column vectors as inputs. Thus, the user may choose to evaluate the swarm in parallel with whatever means.

[sgaure]

Btw, I forgot to mention. The commits also contain a change which allows some upper bounds to be equal to the lower bounds. This can be used as a simple way to keep some variables constant.

[pkofod]

I see, yes I also implemented something similar in NLSolvers.jl. Let me review it. Thanks

[sgaure]

The test failures seem to be unrelated. I came to think of it, should the new parallel keyword be called batched instead? It's not implementing anything parallel.

[github-actions]

Benchmark Results

	master	3d78935...	master / 3d78935...
multivariate/solvers/first_order/AdaMax	0.646 ± 0.0078 ms	0.647 ± 0.008 ms	0.999 ± 0.017
multivariate/solvers/first_order/Adam	0.647 ± 0.008 ms	0.646 ± 0.0081 ms	1 ± 0.018
multivariate/solvers/first_order/BFGS	0.224 ± 0.004 ms	0.223 ± 0.0044 ms	1 ± 0.027
multivariate/solvers/first_order/ConjugateGradient	0.0481 ± 0.0006 ms	0.0479 ± 0.00058 ms	1 ± 0.017
multivariate/solvers/first_order/GradientDescent	1.71 ± 0.014 ms	1.71 ± 0.012 ms	1 ± 0.011
multivariate/solvers/first_order/LBFGS	0.22 ± 0.0057 ms	0.219 ± 0.0036 ms	1 ± 0.031
multivariate/solvers/first_order/MomentumGradientDescent	2.51 ± 0.017 ms	2.51 ± 0.012 ms	0.999 ± 0.0083
multivariate/solvers/first_order/NGMRES	0.554 ± 0.011 ms	0.554 ± 0.011 ms	1 ± 0.027
time_to_load	0.503 ± 0.0047 s	0.502 ± 0.0067 s	1 ± 0.016

Benchmark Plots

A plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR.
Go to "Actions"->"Benchmark a pull request"->[the most recent run]->"Artifacts" (at the bottom).

[pkofod]

I'll look at this later today, it had slipped out of my attention.

[github-actions]

Benchmark Results (Julia vlts)

Time benchmarks

	master	b4f37de...	master / b4f37de...
multivariate/solvers/first_order/AdaMax	0.543 ± 0.0087 ms	0.544 ± 0.01 ms	1 ± 0.024
multivariate/solvers/first_order/Adam	0.542 ± 0.0091 ms	0.541 ± 0.0091 ms	1 ± 0.024
multivariate/solvers/first_order/BFGS	0.261 ± 0.0081 ms	0.26 ± 0.0083 ms	1 ± 0.045
multivariate/solvers/first_order/ConjugateGradient	0.18 ± 0.0031 ms	0.18 ± 0.0031 ms	1 ± 0.024
multivariate/solvers/first_order/GradientDescent	1.54 ± 0.014 ms	1.54 ± 0.011 ms	1 ± 0.012
multivariate/solvers/first_order/LBFGS	0.232 ± 0.0071 ms	0.232 ± 0.0076 ms	1 ± 0.045
multivariate/solvers/first_order/MomentumGradientDescent	2.17 ± 0.015 ms	2.16 ± 0.017 ms	1 ± 0.01
multivariate/solvers/first_order/NGMRES	0.424 ± 0.01 ms	0.425 ± 0.011 ms	0.996 ± 0.035
time_to_load	0.365 ± 0.0017 s	0.366 ± 0.0019 s	0.998 ± 0.0069

Memory benchmarks

	master	b4f37de...	master / b4f37de...
multivariate/solvers/first_order/AdaMax	0.34 k allocs: 7.19 kB	0.34 k allocs: 7.19 kB	1
multivariate/solvers/first_order/Adam	0.34 k allocs: 7.19 kB	0.34 k allocs: 7.19 kB	1
multivariate/solvers/first_order/BFGS	0.336 k allocs: 15 kB	0.336 k allocs: 15 kB	1
multivariate/solvers/first_order/ConjugateGradient	0.338 k allocs: 13.6 kB	0.338 k allocs: 13.6 kB	1
multivariate/solvers/first_order/GradientDescent	1.89 k allocs: 0.0713 MB	1.89 k allocs: 0.0713 MB	1
multivariate/solvers/first_order/LBFGS	0.317 k allocs: 14.2 kB	0.317 k allocs: 14.2 kB	1
multivariate/solvers/first_order/MomentumGradientDescent	2.24 k allocs: 0.077 MB	2.24 k allocs: 0.077 MB	1
multivariate/solvers/first_order/NGMRES	1.41 k allocs: 0.112 MB	1.41 k allocs: 0.112 MB	1
time_to_load	0.153 k allocs: 14.5 kB	0.153 k allocs: 14.5 kB	1

A plot of the benchmark results has been uploaded as an artifact at .

[github-actions]

Benchmark Results (Julia v1)

Time benchmarks

	master	b4f37de...	master / b4f37de...
multivariate/solvers/first_order/AdaMax	0.641 ± 0.0077 ms	0.642 ± 0.0081 ms	1 ± 0.017
multivariate/solvers/first_order/Adam	0.64 ± 0.0076 ms	0.638 ± 0.0083 ms	1 ± 0.018
multivariate/solvers/first_order/BFGS	0.223 ± 0.005 ms	0.223 ± 0.0049 ms	1 ± 0.031
multivariate/solvers/first_order/ConjugateGradient	0.0575 ± 0.00067 ms	0.0575 ± 0.00067 ms	1 ± 0.017
multivariate/solvers/first_order/GradientDescent	1.66 ± 0.01 ms	1.66 ± 0.011 ms	0.999 ± 0.0091
multivariate/solvers/first_order/LBFGS	0.219 ± 0.0039 ms	0.219 ± 0.0045 ms	1 ± 0.027
multivariate/solvers/first_order/MomentumGradientDescent	2.44 ± 0.014 ms	2.44 ± 0.013 ms	0.999 ± 0.0081
multivariate/solvers/first_order/NGMRES	0.537 ± 0.011 ms	0.542 ± 0.011 ms	0.992 ± 0.028
time_to_load	0.415 ± 0.0021 s	0.417 ± 0.002 s	0.995 ± 0.007

Memory benchmarks

	master	b4f37de...	master / b4f37de...
multivariate/solvers/first_order/AdaMax	0.356 k allocs: 7.12 kB	0.356 k allocs: 7.12 kB	1
multivariate/solvers/first_order/Adam	0.356 k allocs: 7.12 kB	0.356 k allocs: 7.12 kB	1
multivariate/solvers/first_order/BFGS	0.306 k allocs: 10.9 kB	0.306 k allocs: 10.9 kB	1
multivariate/solvers/first_order/ConjugateGradient	0.121 k allocs: 4.83 kB	0.121 k allocs: 4.83 kB	1
multivariate/solvers/first_order/GradientDescent	2.15 k allocs: 0.066 MB	2.15 k allocs: 0.066 MB	1
multivariate/solvers/first_order/LBFGS	0.349 k allocs: 12.9 kB	0.349 k allocs: 12.9 kB	1
multivariate/solvers/first_order/MomentumGradientDescent	2.56 k allocs: 0.0754 MB	2.56 k allocs: 0.0754 MB	1
multivariate/solvers/first_order/NGMRES	2.64 k allocs: 0.127 MB	2.64 k allocs: 0.127 MB	1
time_to_load	0.145 k allocs: 11 kB	0.145 k allocs: 11 kB	1

A plot of the benchmark results has been uploaded as an artifact at .