pytorch-optimizer v2.8.0

Change Log

Feature

• Implement A2Grad optimizer, #136
  • Optimal Adaptive and Accelerated Stochastic Gradient Descent
• Implement Accelerated SGD optimizer, #137
  • Accelerating Stochastic Gradient Descent For Least Squares Regression
• Implement Adaptive SGD optimizer, #139
  • Adaptive Gradient Descent without Descent
• Implement SGDW optimizer, #139
  • Decoupled Weight Decay Regularization
• Implement Yogi optimizer, #140
  • Adaptive Methods for Nonconvex Optimization
• Implement SWATS optimizer, #141
  • Improving Generalization Performance by Switching from Adam to SGD
• Implement Fromage optimizer, #142
  • On the distance between two neural networks and the stability of learning
• Implement MSVAG optimizer, #143
  • Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients
• Implement AdaMod optimizer, #144
  • An Adaptive and Momental Bound Method for Stochastic Learning
• Implement AggMo optimizer, #145
  • Aggregated Momentum: Stability Through Passive Damping
• Implement QHAdam, QHM optimizers, #146
  • Quasi-hyperbolic momentum and Adam for deep learning
• Implement PID optimizer, #147
  • A PID Controller Approach for Stochastic Optimization of Deep Networks

Bug

• Fix update in Lion optimizer, #135
• Fix momentum_buffer in SGDP optimizer, #139