pytorch-optimizer v3.4.1

Change Log

Feature

• Support GCSAM optimizer. (#343, #344)
  • Gradient Centralized Sharpness Aware Minimization
  • you can use it from SAM optimizer by setting use_gc=True.
• Support LookSAM optimizer. (#343, #344)
  • Towards Efficient and Scalable Sharpness-Aware Minimization

Update

• Support alternative precision training for Shampoo optimizer. (#339)
• Add more features to and tune Ranger25 optimizer. (#340)
  • AGC + Lookahead variants
  • change default beta1, beta2 to 0.95 and 0.98 respectively
• Skip adding Lookahead wrapper in case of Ranger* optimizers, which already have it in create_optimizer(). (#340)
• Improved optimizer visualization. (#345)
• Rename pytorch_optimizer.optimizer.gc to pytorch_optimizer.optimizer.gradient_centralization to avoid possible conflict with Python built-in function gc. (#349)

Bug

• Fix to update exp_avg_sq after calculating the denominator in ADOPT optimizer. (#346, #347)

Docs

• Update the visualizations. (#340)

Contributions

thanks to @AidinHamedi