Merge pull request #381 from kozistr/release/v3.6.0

[Release] v3.6.0

Author: kozistr

docs/changelogs/v3.6.0.md

@@ -8,8 +8,11 @@
     * [Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension](https://arxiv.org/abs/2502.07752)
 * Implement `VSGD` optimizer. (#377, #378)
     * [Variational Stochastic Gradient Descent for Deep Neural Networks](https://openreview.net/forum?id=xu4ATNjcdy) 
-* Support complex parameters. (#370, #380)
+* Enable training with complex parameters. (#370, #380)
+    * will raise `NoComplexParameterError` for unsupported optimizers, due to its design or not-yet-implemented.
 * Support `maximize` parameter. (#370, #380)
+    * `maximize`: maximize the objective with respect to the params, instead of minimizing.
+* Implement `copy_stochastic()` method. (#381)
 
 ### Update
 
@@ -19,8 +22,13 @@
     * You can still use these variants by passing the parameters to `**kwargs`.
     * Notably, in case of `adanorm` variant, you need to pass `adanorm` (and `adanorm_r` for `r` option) parameter(s) to use this variant, and the name of the state will be changed from `exp_avg_norm` to `exp_avg_adanorm`.
 * Refactor `reset()` to `init_group()` method in the `BaseOptimizer` class. (#380)
-* Refactor `SAM` optimizer faimily. (#380)
+* Refactor `SAM` optimizer family. (#380)
+* Gather `AdamP`, `SGDP` things into `pytorch_optimizer.optimizer.adamp.*`. (#381)
+    * `pytorch_optimizer.optimizer.sgdp.SGDP` to `pytorch_optimizer.optimizer.adamp.SGDP`
+    * `pytorch_optimizer.optimizer.util.projection` to `pytorch_optimizer.optimizer.adamp.projection`
+    * `pytorch_optimizer.optimizer.util.cosine_similarity_by_view` to `pytorch_optimizer.optimizer.adamp.cosine_similarity_by_view`
+* Remove `channel_view()` and `layer_view()` from `pytorch_optimizer.optimizer.util`. (#381)
 
 ### Fix
 
-* Fix shape mismatch issues in the Galore projection for `reverse_std`, `right` and `full` projection types. (#376)
+* Fix shape mismatch issues in the Galore projection for `reverse_std`, `right`, and `full` projection types. (#376)

docs/util.md

@@ -32,26 +32,10 @@
     :docstring:
     :members:
 
-::: pytorch_optimizer.optimizer.utils.channel_view
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.optimizer.utils.layer_view
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.optimizer.utils.cosine_similarity_by_view
-    :docstring:
-    :members:
-
 ::: pytorch_optimizer.optimizer.utils.clip_grad_norm
     :docstring:
     :members:
 
-::: pytorch_optimizer.optimizer.utils.projection
-    :docstring:
-    :members:
-
 ::: pytorch_optimizer.optimizer.utils.unit_norm
     :docstring:
     :members:
@@ -71,3 +55,7 @@
 ::: pytorch_optimizer.optimizer.utils.reg_noise
     :docstring:
     :members:
+
+::: pytorch_optimizer.optimizer.utils.copy_stochastic
+    :docstring:
+    :members:

poetry.lock

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "pytorch_optimizer"
-version = "3.5.1"
+version = "3.6.0"
 description = "optimizer & lr scheduler & objective function collections in PyTorch"
 license = "Apache-2.0"
 authors = ["kozistr <[email protected]>"]

pyproject.toml

@@ -168,6 +168,7 @@
 from pytorch_optimizer.optimizer.utils import (
     CPUOffloadOptimizer,
     clip_grad_norm,
+    copy_stochastic,
     disable_running_stats,
     enable_running_stats,
     get_global_gradient_norm,

pytorch_optimizer/__init__.py

@@ -21,7 +21,7 @@
 from pytorch_optimizer.optimizer.adamax import AdaMax
 from pytorch_optimizer.optimizer.adamg import AdamG
 from pytorch_optimizer.optimizer.adamod import AdaMod
-from pytorch_optimizer.optimizer.adamp import AdamP
+from pytorch_optimizer.optimizer.adamp import SGDP, AdamP
 from pytorch_optimizer.optimizer.adams import AdamS
 from pytorch_optimizer.optimizer.adamw import StableAdamW
 from pytorch_optimizer.optimizer.adan import Adan
@@ -91,7 +91,6 @@
 )
 from pytorch_optimizer.optimizer.scion import SCION, SCIONLight
 from pytorch_optimizer.optimizer.sgd import ASGD, SGDW, VSGD, AccSGD, SGDSaI, SignSGD
-from pytorch_optimizer.optimizer.sgdp import SGDP
 from pytorch_optimizer.optimizer.shampoo import ScalableShampoo, Shampoo
 from pytorch_optimizer.optimizer.sm3 import SM3
 from pytorch_optimizer.optimizer.soap import SOAP