Use autodiff API based on ADTypes instead of symbols

[devmotion]

Fixes #1187

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.45%. Comparing base (af776cf) to head (a79c2a4).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
src/multivariate/optimize/interface.jl	87.50%	1 Missing ⚠️
...tivariate/solvers/constrained/ipnewton/ipnewton.jl	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1195      +/-   ##
==========================================
+ Coverage   87.42%   87.45%   +0.02%     
==========================================
  Files          45       45              
  Lines        3548     3548              
==========================================
+ Hits         3102     3103       +1     
+ Misses        446      445       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[devmotion]

One could consider switching to a different default AD backend (maybe ForwardDiff for univariate optimization and e.g. Mooncake - if at least all tests pass - for multivariate optimization problems? - but the choice for when to switch to which backend is likely also problem/dimension dependent, see also https://docs.sciml.ai/Optimization/stable/API/ad/#ad). But I think it would be better to make such more fundamental changes in a separate PR.

[pkofod]

I agree both to switch to reverse mode for multivariate and forward for univariate, but also that it's probably a separate PR

[github-actions]

Benchmark Results (Julia vlts)

Time benchmarks

	master	a79c2a4...	master / a79c2a4...
multivariate/solvers/first_order/AdaMax	0.543 ± 0.0092 ms	0.615 ± 0.03 ms	0.883 ± 0.045
multivariate/solvers/first_order/Adam	0.543 ± 0.0091 ms	0.543 ± 0.0091 ms	1 ± 0.024
multivariate/solvers/first_order/BFGS	0.262 ± 0.0085 ms	0.262 ± 0.0086 ms	1 ± 0.046
multivariate/solvers/first_order/ConjugateGradient	0.176 ± 0.003 ms	0.204 ± 0.01 ms	0.86 ± 0.045
multivariate/solvers/first_order/GradientDescent	1.55 ± 0.013 ms	1.55 ± 0.012 ms	1 ± 0.011
multivariate/solvers/first_order/LBFGS	0.234 ± 0.008 ms	0.269 ± 0.013 ms	0.869 ± 0.05
multivariate/solvers/first_order/MomentumGradientDescent	2.18 ± 0.015 ms	2.19 ± 0.35 ms	0.996 ± 0.16
multivariate/solvers/first_order/NGMRES	0.434 ± 0.011 ms	0.435 ± 0.011 ms	0.999 ± 0.037
time_to_load	0.432 ± 0.0064 s	0.43 ± 0.0088 s	1 ± 0.025

Memory benchmarks

	master	a79c2a4...	master / a79c2a4...
multivariate/solvers/first_order/AdaMax	0.34 k allocs: 7.16 kB	0.339 k allocs: 7.09 kB	1.01
multivariate/solvers/first_order/Adam	0.34 k allocs: 7.16 kB	0.339 k allocs: 7.09 kB	1.01
multivariate/solvers/first_order/BFGS	0.36 k allocs: 15.5 kB	0.359 k allocs: 15.5 kB	1
multivariate/solvers/first_order/ConjugateGradient	0.362 k allocs: 14.2 kB	0.361 k allocs: 14.1 kB	1
multivariate/solvers/first_order/GradientDescent	2.09 k allocs: 0.0759 MB	2.08 k allocs: 0.0758 MB	1
multivariate/solvers/first_order/LBFGS	0.341 k allocs: 14.7 kB	0.34 k allocs: 14.7 kB	1
multivariate/solvers/first_order/MomentumGradientDescent	2.44 k allocs: 0.0815 MB	2.44 k allocs: 0.0815 MB	1
multivariate/solvers/first_order/NGMRES	1.56 k allocs: 0.117 MB	1.56 k allocs: 0.117 MB	1
time_to_load	0.153 k allocs: 14.5 kB	0.153 k allocs: 14.5 kB	1

A plot of the benchmark results has been uploaded as an artifact at .

[pkofod]

JET complains

[devmotion]

I checked it locally, and none of the things JET complains are introduced or changed by the PR. It seems the PR just helps JET to perform its type-signature-based analysis more thoroughly, and hence more problems are revealed 😄

I could fix a few of them in this PR but arguably it might be better to do that in a separate PR.

[devmotion]

#1209 fixes the JET errors on the master branch.