Use autodiff API based on ADTypes instead of symbols

[devmotion]

Fixes #1187

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.59%. Comparing base (c3ba21c) to head (9ab4a83).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1195   +/-   ##
=======================================
  Coverage   87.59%   87.59%           
=======================================
  Files          45       45           
  Lines        3515     3515           
=======================================
  Hits         3079     3079           
  Misses        436      436           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[devmotion]

One could consider switching to a different default AD backend (maybe ForwardDiff for univariate optimization and e.g. Mooncake - if at least all tests pass - for multivariate optimization problems? - but the choice for when to switch to which backend is likely also problem/dimension dependent, see also https://docs.sciml.ai/Optimization/stable/API/ad/#ad). But I think it would be better to make such more fundamental changes in a separate PR.

[pkofod]

I agree both to switch to reverse mode for multivariate and forward for univariate, but also that it's probably a separate PR

[github-actions]

Benchmark Results (Julia vlts)

Time benchmarks

	master	9ab4a83...	master / 9ab4a83...
multivariate/solvers/first_order/AdaMax	0.545 ± 0.009 ms	0.546 ± 0.009 ms	0.998 ± 0.023
multivariate/solvers/first_order/Adam	0.543 ± 0.0092 ms	0.545 ± 0.009 ms	0.996 ± 0.024
multivariate/solvers/first_order/BFGS	0.261 ± 0.0082 ms	0.262 ± 0.008 ms	0.996 ± 0.043
multivariate/solvers/first_order/ConjugateGradient	0.174 ± 0.0031 ms	0.174 ± 0.0033 ms	1 ± 0.026
multivariate/solvers/first_order/GradientDescent	1.54 ± 0.012 ms	1.54 ± 0.01 ms	0.998 ± 0.01
multivariate/solvers/first_order/LBFGS	0.233 ± 0.0075 ms	0.232 ± 0.0073 ms	1 ± 0.045
multivariate/solvers/first_order/MomentumGradientDescent	2.16 ± 0.016 ms	2.17 ± 0.012 ms	0.995 ± 0.0093
multivariate/solvers/first_order/NGMRES	0.429 ± 0.011 ms	0.428 ± 0.01 ms	1 ± 0.035
time_to_load	0.415 ± 0.0049 s	0.425 ± 0.0036 s	0.976 ± 0.014

Memory benchmarks

	master	9ab4a83...	master / 9ab4a83...
multivariate/solvers/first_order/AdaMax	0.34 k allocs: 7.16 kB	0.339 k allocs: 7.09 kB	1.01
multivariate/solvers/first_order/Adam	0.34 k allocs: 7.16 kB	0.339 k allocs: 7.09 kB	1.01
multivariate/solvers/first_order/BFGS	0.336 k allocs: 15 kB	0.335 k allocs: 14.9 kB	1
multivariate/solvers/first_order/ConjugateGradient	0.332 k allocs: 13.5 kB	0.331 k allocs: 13.4 kB	1
multivariate/solvers/first_order/GradientDescent	1.89 k allocs: 0.0713 MB	1.89 k allocs: 0.0712 MB	1
multivariate/solvers/first_order/LBFGS	0.317 k allocs: 14.2 kB	0.316 k allocs: 14.1 kB	1
multivariate/solvers/first_order/MomentumGradientDescent	2.24 k allocs: 0.077 MB	2.23 k allocs: 0.0769 MB	1
multivariate/solvers/first_order/NGMRES	1.51 k allocs: 0.117 MB	1.51 k allocs: 0.117 MB	1
time_to_load	0.153 k allocs: 14.5 kB	0.153 k allocs: 14.5 kB	1

A plot of the benchmark results has been uploaded as an artifact at .