pytorch-optimizer v3.6.1

Change Log

Feature

• Implement more cooldown types for WSD learning rate scheduler. (#382, #386)
• Implement AdamWSN optimizer. (#387, #389)
  • Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees
• Implement AdamC optimizer. (#388, #390)
  • Why Gradients Rapidly Increase Near the End of Training

Update

• Change the default range of the beta parameter from [0, 1] to [0, 1). (#392)

Fix

• Fix to use momentum buffer instead of the gradient to calculate LMO. (#385)