pytorch-optimizer v3.4.0

Change Log

Feature

• Implement FOCUS optimizer. (#330, #331)
  • First Order Concentrated Updating Scheme
• Implement PSGD Kron optimizer. (#336, #337)
  • preconditioned stochastic gradient descent w/ Kron pre-conditioner
• Implement EXAdam optimizer. (#338, #339)
  • The Power of Adaptive Cross-Moments

Update

• Support OrthoGrad variant to Ranger25. (#332)
  • Ranger25 optimizer is my experimental-crafted optimizer, which mixes lots of optimizer variants such as ADOPT + AdEMAMix + Cautious + StableAdamW + Adam-Atan2 + OrthoGrad.

Fix

• Add the missing state property in OrthoGrad optimizer. (#326, #327)
• Add the missing state_dict, and load_state_dict methods to TRAC and OrthoGrad optimizers. (#332)
• Skip when the gradient is sparse in OrthoGrad optimizer. (#332)
• Support alternative precision training in SOAP optimizer. (#333)
• Store SOAP condition matrices as the dtype of their parameters. (#335)

Contributions

thanks to @Vectorrent, @kylevedder

pytorch-optimizer v3.3.4

Change Log

Feature

• Support OrthoGrad feature for create_optimizer(). (#324)
• Enhanced flexibility for the optimizer parameter in Lookahead, TRAC, and OrthoGrad optimizers. (#324)
  • Now supports both torch.optim.Optimizer instances and classes
  • You can now use Lookahead optimizer in two ways.
    • Lookahead(AdamW(model.parameters(), lr=1e-3), k=5, alpha=0.5)
    • Lookahead(AdamW, k=5, alpha=0.5, params=model.parameters())
• Implement SPAM optimizer. (#324)
  • Spike-Aware Adam with Momentum Reset for Stable LLM Training
• Implement TAM, and AdaTAM optimizers. (#325)
  • Torque-Aware Momentum

pytorch-optimizer v3.3.3

Change Log

Feature

• Implement Grams optimizer. (#317, #318)
  • Grams: Gradient Descent with Adaptive Momentum Scaling
• Support stable_adamw variant for ADOPT and AdEMAMix optimizer. (#321)
  • optimizer = ADOPT(model.parameters(), ..., stable_adamw=True)
• Implement an experimental optimizer Ranger25 (not tested). (#321)
  • mixing ADOPT + AdEMAMix + StableAdamW + Cautious + RAdam optimizers.
• Implement OrthoGrad optimizer. (#321)
  • Grokking at the Edge of Numerical Stability
• Support Adam-Atan2 feature for Prodigy optimizer when eps is None. (#321)
  • Scaling Exponents Across Parameterizations and Optimizers

pytorch-optimizer v3.3.2

Change Log

Feature

• Implement SGDSaI optimizer. (#315, #316)
  • No More Adam: Learning Rate Scaling at Initialization is All You Need

Bug

• Clone exp_avg before calling apply_cautious not to mask exp_avg. (#316)

pytorch-optimizer v3.3.1

Change Log

Feature

• Support Cautious variant to AdaShift optimizer. (#310)
• Save the state of the Lookahead optimizer too. (#310)
• Implement APOLLO optimizer. (#311, #312)
  • SGD-like Memory, AdamW-level Performance
• Rename the Apollo (An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization) optimizer name to ApolloDQN not to overlap with the new optimizer name APOLLO. (#312)
• Implement MARS optimizer. (#313, #314)
  • Unleashing the Power of Variance Reduction for Training Large Models
• Support Cautious variant to MARS optimizer. (#314)

Bug

• Fix bias_correction in AdamG optimizer. (#305, #308)
• Fix a potential bug when loading the state for Lookahead optimizer. (#306, #310)

Docs

• Add more visualizations. (#310, #314)

Contributions

thanks to @Vectorrent

pytorch-optimizer v3.3.0

Change Log

Feature

• Support PaLM variant for ScheduleFreeAdamW optimizer. (#286, #288)
  • you can use this feature by setting use_palm to True.
• Implement ADOPT optimizer. (#289, #290)
  • Modified Adam Can Converge with Any β2 with the Optimal Rate
• Implement FTRL optimizer. (#291)
  • Follow The Regularized Leader
• Implement Cautious optimizer feature. (#294)
  • Improving Training with One Line of Code
  • you can use it by setting cautious=True for Lion, AdaFactor and AdEMAMix optimizers.
• Improve the stability of ADOPT optimizer. (#294)
  • Note
• Support a new projection type random for GaLoreProjector. (#294)
• Implement DeMo optimizer. (#300, #301)
  • Decoupled Momentum Optimization
• Implement Muon optimizer. (#302)
  • MomentUm Orthogonalized by Newton-schulz
• Implement ScheduleFreeRAdam optimizer. (#304)
• Implement LaProp optimizer. (#304)
  • Separating Momentum and Adaptivity in Adam
• Support Cautious variant to LaProp, AdamP, Adopt optimizers. (#304).

Refactor

• Big refactoring, removing direct import from pytorch_optimizer.*.
  • I removed some methods not to directly import from it from pytorch_optimzier.* because they're probably not used frequently and actually not an optimizer rather utils only used for specific optimizers.
  • pytorch_optimizer.[Shampoo stuff] -> pytorch_optimizer.optimizers.shampoo_utils.[Shampoo stuff].
    • shampoo_utils like Graft, BlockPartitioner, PreConditioner, etc. You can check the details here.
  • pytorch_optimizer.GaLoreProjector -> pytorch_optimizer.optimizers.galore.GaLoreProjector.
  • pytorch_optimizer.gradfilter_ema -> pytorch_optimizer.optimizers.grokfast.gradfilter_ema.
  • pytorch_optimizer.gradfilter_ma -> pytorch_optimizer.optimizers.grokfast.gradfilter_ma.
  • pytorch_optimizer.l2_projection -> pytorch_optimizer.optimizers.alig.l2_projection.
  • pytorch_optimizer.flatten_grad -> pytorch_optimizer.optimizers.pcgrad.flatten_grad.
  • pytorch_optimizer.un_flatten_grad -> pytorch_optimizer.optimizers.pcgrad.un_flatten_grad.
  • pytorch_optimizer.reduce_max_except_dim -> pytorch_optimizer.optimizers.sm3.reduce_max_except_dim.
  • pytorch_optimizer.neuron_norm -> pytorch_optimizer.optimizers.nero.neuron_norm.
  • pytorch_optimizer.neuron_mean -> pytorch_optimizer.optimizers.nero.neuron_mean.

Docs

• Add more visualizations. (#297)

Bug

• Add optimizer parameter to PolyScheduler constructor. (#295)

Contributions

thanks to @tanganke

pytorch-optimizer v3.2.0

Change Log

Feature

• Implement SOAP optimizer. (#275)
  • SOAP: Improving and Stabilizing Shampoo using Adam
• Support AdEMAMix variants. (#276)
  • bnb_ademamix8bit, bnb_ademamix32bit, bnb_paged_ademamix8bit, bnb_paged_ademamix32bit
• Support 8/4bit, fp8 optimizers. (#208, #281)
  • torchao_adamw8bit, torchao_adamw4bit, torchao_adamwfp8.
• Support a module-name-level (e.g. LayerNorm) weight decay exclusion for get_optimizer_parameters. (#282, #283)
• Implement CPUOffloadOptimizer, which offloads optimizer to CPU for single-GPU training. (#284)
• Support a regex-based filter for searching names of optimizers, lr schedulers, and loss functions.

Bug

• Fix should_grokfast condition when initialization. (#279, #280)

Contributions

thanks to @Vectorrent

pytorch-optimizer v3.1.2

Change Log

Feature

• Implement AdEMAMix optimizer. (#272)
  • THE ADEMAMIX OPTIMIZER: BETTER, FASTER, OLDER

Bug

• Add **kwargs to the parameters for dummy placeholder. (#270, #271)

pytorch-optimizer v3.1.1

Change Log

Feature

• Implement TRAC optimizer. (#263)
  • Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning
• Support AdamW optimizer via create_optimizer(). (#263)
• Implement AdamG optimizer. (#264, #265)
  • Towards Stability of Parameter-free Optimization

Bug

• Handle the optimizers that only take the model instead of the parameters in create_optimizer(). (#263)
• Move the variable to the same device with the parameter. (#266, #267)

pytorch-optimizer v3.1.0

Change Log

Feature

• Implement AdaLomo optimizer. (#258)
  • Low-memory Optimization with Adaptive Learning Rate
• Support Q-GaLore optimizer. (#258)
  • Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.
  • you can use by optimizer = load_optimizer('q_galore_adamw8bit')
• Support more bnb optimizers. (#258)
  • bnb_paged_adam8bit, bnb_paged_adamw8bit, bnb_*_*32bit.
• Improve power_iteration() speed up to 40%. (#259)
• Improve reg_noise() (E-MCMC) speed up to 120%. (#260)
• Support disable_lr_scheduler parameter for Ranger21 optimizer to disable built-in learning rate scheduler. (#261)

Refactor

• Refactor AdamMini optimizer. (#258)
• Deprecate optional dependency, bitsandbytes. (#258)
• Move get_rms, approximate_sq_grad functions to BaseOptimizer for reusability. (#258)
• Refactor shampoo_utils.py. (#259)
• Add debias, debias_adam methods in BaseOptimizer. (#261)
• Refactor to use BaseOptimizer only, not inherit multiple classes. (#261)

Bug

• Fix several bugs in AdamMini optimizer. (#257)

Contributions

thanks to @sdbds