Merge pull request #163 from kozistr/refactor/codes

[Refactor] codes

Author: kozistr

.github/workflows/publish.yml

@@ -20,6 +20,7 @@ jobs:
         with:
           tag_name: ${{ github.ref }}
           release_name: pytorch-optimizer ${{ github.ref }}
+          body_path: docs/changelog/${{ github.ref }}.md
           draft: false
           prerelease: false
   deploy:

docs/changelogs/v2.7.0.md

@@ -0,0 +1,41 @@
+## Change Log
+
+### Feature
+
+* Implement `AdaNorm` optimizer (#133)
+  * [AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs](https://arxiv.org/abs/2210.06364)
+* Implement `RotoGrad` optimizer (#124, #134)
+  * [RotoGrad: Gradient Homogenization in Multitask Learning](https://arxiv.org/abs/2103.02631)
+* Implement `D-Adapt Adan` optimizer (#134)
+* Support `AdaNorm` variant (#133, #134) 
+  * AdaBelief
+  * AdamP
+  * AdamS
+  * AdaPNM
+  * diffGrad
+  * Lamb
+  * RAdam
+  * Ranger
+  * Adan
+* Support `AMSGrad` variant (#133, #134)
+  * diffGrad
+  * AdaFactor
+* Support `degenerated_to_sgd` (#133)
+  * Ranger
+  * Lamb
+  
+### Refactor
+
+* Rename `adamd_debias_term` to `adam_debias` (#133)
+* Merge the rectified version with the original (#133)
+  * diffRGrad + diffGrad -> diffGrad 
+  * RaLamb + Lamb -> Lamb
+  * now you can simply use with `rectify=True`
+ 
+### Bug
+
+* Fix `previous_grad` deepcopy issue in Adan optimizer (#134)
+
+### Diff
+
+[2.6.1...2.7.0](https://github.com/kozistr/pytorch_optimizer/compare/v2.6.1...v2.7.0)

docs/changelogs/v2.8.0.md

@@ -0,0 +1,37 @@
+## Change Log
+
+### Feature
+
+* Implement A2Grad optimizer (#136)
+  * [Optimal Adaptive and Accelerated Stochastic Gradient Descent](https://arxiv.org/abs/1810.00553)
+* Implement Accelerated SGD optimizer (#137)
+  * [Accelerating Stochastic Gradient Descent For Least Squares Regression](https://arxiv.org/abs/1704.08227)
+* Implement Adaptive SGD optimizer (#139)
+  * [Adaptive Gradient Descent without Descent](https://arxiv.org/abs/1910.09529)
+* Implement SGDW optimizer (#139)
+  * [Decoupled Weight Decay Regularization](https://arxiv.org/abs/1711.05101)
+* Implement Yogi optimizer (#140)
+  * [Adaptive Methods for Nonconvex Optimization](https://papers.nips.cc/paper_files/paper/2018/hash/90365351ccc7437a1309dc64e4db32a3-Abstract.html)
+* Implement SWATS optimizer (#141)
+  * [Improving Generalization Performance by Switching from Adam to SGD](https://arxiv.org/abs/1712.07628) 
+* Implement Fromage optimizer (#142)
+  * [On the distance between two neural networks and the stability of learning](https://arxiv.org/abs/2002.03432) 
+* Implement MSVAG optimizer (#143)
+  * [Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients](https://arxiv.org/abs/1705.07774) 
+* Implement AdaMod optimizer (#144)
+  * [An Adaptive and Momental Bound Method for Stochastic Learning](https://arxiv.org/abs/1910.12249) 
+* Implement AggMo optimizer (#145)
+  * [Aggregated Momentum: Stability Through Passive Damping](https://arxiv.org/abs/1804.00325)
+* Implement QHAdam, QHM optimizers (#146)
+  * [Quasi-hyperbolic momentum and Adam for deep learning](https://arxiv.org/abs/1810.06801)
+* Implement PID optimizer (#147)
+  * [A PID Controller Approach for Stochastic Optimization of Deep Networks](http://www4.comp.polyu.edu.hk/~cslzhang/paper/CVPR18_PID.pdf) 
+
+### Bug
+
+* Fix `update` in Lion optimizer (#135)
+* Fix `momentum_buffer` in SGDP optimizer (#139)
+
+### Diff
+
+[2.7.0...2.8.0](https://github.com/kozistr/pytorch_optimizer/compare/v2.7.0...v2.8.0)

docs/changelogs/v2.9.0.md

@@ -0,0 +1,36 @@
+## Change Log
+
+### Feature
+
+* Implement AdaMax optimizer (#148)
+  * A variant of Adam based on the infinity norm
+* Implement Gravity optimizer (#151)
+  * [a Kinematic Approach on Optimization in Deep Learning](https://arxiv.org/abs/2101.09192)
+* Implement AdaSmooth optimizer (#153)
+  * [An Adaptive Learning Rate Method based on Effective Ratio](https://arxiv.org/abs/2204.00825v1)
+* Implement SRMM optimizer (#154)
+  * [Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates](https://arxiv.org/abs/2201.01652)
+* Implement AvaGrad optimizer (#155) 
+  * [Domain-independent Dominance of Adaptive Methods](https://arxiv.org/abs/1912.01823)
+* Implement AdaShift optimizer (#157) 
+  * [Decorrelation and Convergence of Adaptive Learning Rate Methods](https://arxiv.org/abs/1810.00143v4)
+* Upgrade to D-Adaptation v3 (#158, #159)
+* Implement AdaDelta optimizer (#160)
+  * [An Adaptive Learning Rate Method](https://arxiv.org/abs/1212.5701v1) 
+
+### Docs
+
+* Fix readthedocs build issue (#156)
+* Move citations into table (#156) 
+
+### Refactor
+
+* Refactor validation logic (#149, #150)
+* Rename `amsbound`, `amsgrad` terms into `ams_bound` (#149)
+* Return gradient instead of the parameter, AGC. (#149)
+* Refactor duplicates (e.g. rectified step size, AMSBound, AdamD, AdaNorm, weight decay) into re-usable functions (#150)
+* Move `pytorch_optimizer.experimental` under `pytorch_optimizer.*.experimental`
+
+### Diff
+
+[2.8.0...2.9.0](https://github.com/kozistr/pytorch_optimizer/compare/v2.8.0...v2.9.0)

pytorch_optimizer/optimizer/adai.py

@@ -103,7 +103,7 @@ def step(self, closure: CLOSURE = None) -> LOSS:
                 state['step'] += 1
 
                 if self.use_gc:
-                    grad = centralize_gradient(grad, gc_conv_only=False)
+                    centralize_gradient(grad, gc_conv_only=False)
 
                 bias_correction2: float = 1.0 - beta2 ** state['step']
 

pytorch_optimizer/optimizer/adamp.py

@@ -122,7 +122,7 @@ def step(self, closure: CLOSURE = None) -> LOSS:
                         state['exp_grad_norm'] = torch.zeros((1,), dtype=grad.dtype, device=grad.device)
 
                 if self.use_gc:
-                    grad = centralize_gradient(grad, gc_conv_only=False)
+                    centralize_gradient(grad, gc_conv_only=False)
 
                 s_grad = self.get_adanorm_gradient(
                     grad=grad,

pytorch_optimizer/optimizer/adan.py

@@ -130,7 +130,7 @@ def step(self, closure: CLOSURE = None) -> LOSS:
                 grad.mul_(clip_global_grad_norm)
 
                 if self.use_gc:
-                    grad = centralize_gradient(grad, gc_conv_only=False)
+                    centralize_gradient(grad, gc_conv_only=False)
 
                 grad_diff = state['previous_grad']
                 grad_diff.add_(grad)

pytorch_optimizer/optimizer/gc.py

@@ -1,14 +1,12 @@
 import torch
 
 
-def centralize_gradient(x: torch.Tensor, gc_conv_only: bool = False) -> torch.Tensor:
+def centralize_gradient(x: torch.Tensor, gc_conv_only: bool = False):
     r"""Gradient Centralization (GC).
 
     :param x: torch.Tensor. gradient.
     :param gc_conv_only: bool. 'False' for both conv & fc layers.
-    :return: torch.Tensor. centralized gradient.
     """
     size: int = x.dim()
     if (gc_conv_only and size > 3) or (not gc_conv_only and size > 1):
         x.add_(-x.mean(dim=tuple(range(1, size)), keepdim=True))
-    return x

pytorch_optimizer/optimizer/lion.py

@@ -90,7 +90,7 @@ def step(self, closure: CLOSURE = None) -> LOSS:
                         state['exp_grad_norm'] = torch.zeros((1,), dtype=grad.dtype, device=grad.device)
 
                 if self.use_gc:
-                    grad = centralize_gradient(grad, gc_conv_only=False)
+                    centralize_gradient(grad, gc_conv_only=False)
 
                 self.apply_weight_decay(
                     p=p,

pytorch_optimizer/optimizer/ranger.py

@@ -140,7 +140,7 @@ def step(self, closure: CLOSURE = None) -> LOSS:
                         state['exp_grad_norm'] = torch.zeros((1,), dtype=grad.dtype, device=grad.device)
 
                 if self.use_gc and grad.dim() > self.gc_gradient_threshold:
-                    grad = centralize_gradient(grad, gc_conv_only=False)
+                    centralize_gradient(grad, gc_conv_only=False)
 
                 self.apply_weight_decay(
                     p=p,