pytorch-optimizer v3.0.2

Change Log

Feature

• Implement WSD LR Scheduler. (#247, #248)
  • Warmup-Stable-Decay LR Scheduler
• Add more Pytorch built-in lr schedulers. (#248)
• Implement Kate optimizer. (#249, #251)
  • Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad
• Implement StableAdamW optimizer. (#250, #252)
  • Stable and low-precision training for large-scale vision-language models
• Implement AdamMini optimizer. (#246, #253)
  • Use Fewer Learning Rates To Gain More

Refactor

• Refactor Chebyschev lr scheduler modules. (#248)
  • Rename get_chebyshev_lr to get_chebyshev_lr_lambda.
  • Rename get_chebyshev_schedule to get_chebyshev_perm_steps.
  • Call get_chebyshev_schedule function to get LamdbaLR scheduler object.
• Refactor with ScheduleType. (#248)

pytorch-optimizer v3.0.1

Change Log

Feature

• Implement FAdam optimizer. (#241, #242)
  • Adam is a natural gradient optimizer using diagonal empirical Fisher information
• Tweak AdaFactor optimizer. (#236, #243)
  • support not-using-first-momentum when beta1 is not given
  • default dtype for first momentum to bfloat16
  • clip second momentum to 0.999
• Implement GrokFast optimizer. (#244, #245)
  • Accelerated Grokking by Amplifying Slow Gradients

Bug

• Wrong typing of reg_noise. (#239, #240)
• Lookahead`s param_groups attribute is not loaded from checkpoint. (#237, #238)

Contributions

thanks to @michaldyczko

pytorch-optimizer v3.0.0

Change Log

The major version is updated! (v2.12.0 -> v3.0.0) (#164)

Many optimizers, learning rate schedulers, and objective functions are in pytorch-optimizer.
Currently, pytorch-optimizer supports 67 optimizers (+ bitsandbytes), 11 lr schedulers, and 13 loss functions, and reached about 4 ~ 50K downloads / month (peak is 75K downloads / month)!

The reason for updating the major version from v2 to v3 is that I think it's a good time to ship the recent implementations (the last update was about 7 months ago) and plan to pivot to new concepts like training utilities while maintaining the original features (e.g. optimizers).
Also, rich test cases, benchmarks, and examples are on the list!

Finally, thanks for using the pytorch-optimizer, and feel free to make any requests :)

Feature

• Implement REX lr scheduler. (#217, #222)
  • Revisiting Budgeted Training with an Improved Schedule
• Implement Aida optimizer. (#220, #221)
  • A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range
• Implement WSAM optimizer. (#213, #216)
  • Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term
• Implement GaLore optimizer. (#224, #228)
  • Memory-Efficient LLM Training by Gradient Low-Rank Projection
• Implement Adalite optimizer. (#225, #229)
• Implement bSAM optimizer. (#212, #233)
  • SAM as an Optimal Relaxation of Bayes
• Implement Schedule-Free optimizer. (#230, #233)
  • Schedule-Free optimizers
• Implement EMCMC. (#231, #233)
  • Entropy-MCMC: Sampling from flat basins with ease

Fix

• Fix SRMM to allow operation beyond memory_length. (#227)

Dependency

• Drop Python 3.7 support officially. (#221)
  • Please check the README.
• Update bitsandbytes to 0.43.0. (#228)

Docs

• Add missing parameters in Ranger21 optimizer document. (#214, #215)
• Fix WSAM optimizer paper link. (#219)

Contributions

thanks to @sdbds, @i404788

Diff

• from the previous major version : 2.0.0...3.0.0
• from the previous version: 2.12.0...3.0.0

pytorch-optimizer v2.12.0

Change Log

Feature

• Support bitsandbytes optimizer. (#211)
  • now, you can install with pip3 install pytorch-optimizer[bitsandbytes]
  • supports 8 bnb optimizers.
    • bnb_adagrad8bit, bnb_adam8bit, bnb_adamw8bit, bnb_lion8bit, bnb_lamb8bit, bnb_lars8bit, bnb_rmsprop8bit, bnb_sgd8bit.

Docs

• Introduce mkdocs with material theme. (#204, #206)
  • documentation : https://pytorch-optimizers.readthedocs.io/en/latest/

Diff

2.11.2...2.12.0

pytorch-optimizer v2.11.2

Change Log

Feature

• Implement DAdaptLion optimizer (#203)
  • Lion with D-Adaptation

Fix

• Fix Lookahead optimizer (#200, #201, #202)
  • When using PyTorch Lightning which expects your optimiser to be a subclass of Optimizer.
• Fix default rectify to False in AdaBelief optimizer (#203)

Test

• Add DynamicLossScaler test case

Docs

• Highlight the code blocks
• Fix pepy badges

Contributions

thanks to @georg-wolflein

Diff

2.11.1...2.11.2

pytorch-optimizer v2.11.1

Change Log

Feature

• Implement Tiger optimizer (#192)
  • A Tight-fisted Optimizer
• Implement CAME optimizer (#196)
  • Confidence-guided Adaptive Memory Efficient Optimization
• Implement loss functions (#198)
  • Tversky Loss : Tversky loss function for image segmentation using 3D fully convolutional deep networks
  • Focal Tversky Loss
  • Lovasz Hinge Loss : The Lovász-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks

Diff

2.11.0...2.11.1

pytorch-optimizer v2.11.0

Change Log

Feature

• Implement PAdam optimizer (#186)
  • Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Implement LOMO optimizer (#188)
  • Full Parameter Fine-tuning for Large Language Models with Limited Resources
• Implement loss functions (#189)
  • BCELoss
  • BCEFocalLoss
  • FocalLoss : Focal Loss for Dense Object Detection
  • FocalCosineLoss : Data-Efficient Deep Learning Method for Image Classification Using Data Augmentation, Focal Cosine Loss, and Ensemble
  • DiceLoss : Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
  • LDAMLoss : Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
  • JaccardLoss
  • BiTemperedLogisticLoss : Robust Bi-Tempered Logistic Loss Based on Bregman Divergences

Diff

2.10.1...2.11.0

pytorch-optimizer v2.10.1

Change Log

Feature

• Implement Prodigy optimizer (#183)
  • An Expeditiously Adaptive Parameter-Free Learner

Fix

• perturb isn't multiplied by -step_size in SWATS optimizer. (#179)
• chebyshev step has size of T while the permutation is 2^T. (#168, #181)

Diff

2.10.0...2.10.1

pytorch-optimizer v2.10.0

Change Log

Feature

• Implement Amos optimizer (#174)
  • An Adam-style Optimizer with Adaptive Weight Decay towards Model-Oriented Scale
• Implement SignSGD optimizer (#176)
  • Compressed Optimisation for Non-Convex Problems
• Implement AdaHessian optimizer (#176)
  • An Adaptive Second Order Optimizer for Machine Learning
• Implement SophiaH optimizer (#173, #176)
  • A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
• Implement re-usable functions to compute hessian in BaseOptimizer (#176, #177)
  • two types of distribution are supported (Gaussian, Rademacher).
• Support AdamD feature for AdaHessian optimizer (#177)

Diff

2.9.1...2.10.0

Contributions

thanks to @i404788

pytorch-optimizer v2.9.1

Change Log

Fix

• fix weight decay in Ranger21 (#170)

Diff

2.9.0...2.9.1