pytorch-optimizer v3.3.3

Change Log

Feature

• Implement Grams optimizer. (#317, #318)
  • Grams: Gradient Descent with Adaptive Momentum Scaling
• Support stable_adamw variant for ADOPT and AdEMAMix optimizer. (#321)
  • optimizer = ADOPT(model.parameters(), ..., stable_adamw=True)
• Implement an experimental optimizer Ranger25 (not tested). (#321)
  • mixing ADOPT + AdEMAMix + StableAdamW + Cautious + RAdam optimizers.
• Implement OrthoGrad optimizer. (#321)
  • Grokking at the Edge of Numerical Stability
• Support Adam-Atan2 feature for Prodigy optimizer when eps is None. (#321)
  • Scaling Exponents Across Parameterizations and Optimizers