pytorch-optimizer v2.10.0

Change Log

Feature

• Implement Amos optimizer (#174)
  • An Adam-style Optimizer with Adaptive Weight Decay towards Model-Oriented Scale
• Implement SignSGD optimizer (#176)
  • Compressed Optimisation for Non-Convex Problems
• Implement AdaHessian optimizer (#176)
  • An Adaptive Second Order Optimizer for Machine Learning
• Implement SophiaH optimizer (#173, #176)
  • A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
• Implement re-usable functions to compute hessian in BaseOptimizer (#176, #177)
  • two types of distribution are supported (Gaussian, Rademacher).
• Support AdamD feature for AdaHessian optimizer (#177)

Diff

2.9.1...2.10.0

Contributions

thanks to @i404788