pytorch-optimizer v3.5.0

Change Log

Feature

• Support StableSPAM optimizer. (#358, #359)
  • How to Train in 4-Bit More Stably than 16-Bit Adam
• Support ScheduleFreeWrapper. (#334, #360)
• Implement AdaGC optimizer. (#364, #366)
  • Improving Training Stability for Large Language Model Pretraining
• Implement Simplified-Ademamix optimizer. (#364, #366)
  • Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants
• Support Ackley function for testing optimization algorithms.

Update

• Update Muon optimizer. (#355, #356)
  • support decoupled weight decay.
  • adjust default hyperparameters the same as the original implementation.
  • support adjusted lr from the Moonlight. you can use it by setting use_adjusted_lr=True.
• Tune the performance of the coupled Newton iteration method by 5% increase. (#360)
• Update SCION optimizer. (#361)
  • add scale parameter.
  • update get_lmo_direction.

Fix

• bias_correction2 in ScheduleFreeRAdam optimizer. (#354)
• potential bug in SPAM optimizer. (#365)
• initialize the z state within the step() of the ScheduleFreeWrapper. (#363, #366)