pytorch-optimizer v2.7.0

Change Log

Features

• Implement AdaNorm optimizer, #133
  • AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs
• Implement RotoGrad optimizer, #124, #134
  • RotoGrad: Gradient Homogenization in Multitask Learning
• Implement D-Adapt Adan optimizer, #134
• Support AdaNorm variant, #133, #134
  • AdaBelief
  • AdamP
  • AdamS
  • AdaPNM
  • diffGrad
  • Lamb
  • RAdam
  • Ranger
  • Adan
• Support AMSGrad variant, #133, #134
  • diffGrad
  • AdaFactor
• Support degenerated_to_sgd, #133
  • Ranger
  • Lamb

Refactor

• Rename adamd_debias_term to adam_debias, #133
• Merge the rectified version with the original, #133
  • diffRGrad + diffGrad -> diffGrad
  • RaLamb + Lamb -> Lamb
  • now you can simply use with rectify=True

Fix

• Fix previous_grad deepcopy issue in Adan optimizer. #134