pytorch-optimizer v2.9.0

Change Log

Feature

• Implement AdaMax optimizer, #148
  • A variant of Adam based on the infinity norm
• Implement Gravity optimizer, #151
  • a Kinematic Approach on Optimization in Deep Learning
• Implement AdaSmooth optimizer, #153
  • An Adaptive Learning Rate Method based on Effective Ratio
• Implement SRMM optimizer, #154
  • Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates
• Implement AvaGrad optimizer, #155
  • Domain-independent Dominance of Adaptive Methods
• Implement AdaShift optimizer, #157
  • Decorrelation and Convergence of Adaptive Learning Rate Methods
• Upgrade to D-Adaptation v3, #158, #159
• Implement AdaDelta optimizer, #160
  • An Adaptive Learning Rate Method

Docs

• Fix readthedocs build issue, #156
• Move citations into table, #156

Refactor

• Refactor validation logic, #149, #150
• Rename amsbound, amsgrad terms into ams_bound, #149
• Return gradient instead of the parameter, AGC. #149
• Refactor duplicates (e.g. rectified step size, AMSBound, AdamD, AdaNorm, weight decay) into re-usable functions, #150
• Move pytorch_optimizer.experimental under pytorch_optimizer.*.experimental

Diff

2.8.0...2.9.0