Merge pull request #297 from kozistr/update/codes

[Docs] document stuff

Author: kozistr

README.md

@@ -355,6 +355,10 @@ With the default bias-correction, Adam may actually make larger than requested g
 
 Correcting the norm of a gradient in each iteration based on the adaptive training history of gradient norm.
 
+### Cautious optimizer
+
+Updates only occur when the proposed update direction aligns with the current gradient.
+
 ## Frequently asked questions
 
 [here](docs/qa.md)

docs/changelogs/v3.3.0.md

@@ -30,3 +30,15 @@
     * `pytorch_optimizer.reduce_max_except_dim` -> `pytorch_optimizer.optimizers.sm3.reduce_max_except_dim`.
     * `pytorch_optimizer.neuron_norm` -> `pytorch_optimizer.optimizers.nero.neuron_norm`.
     * `pytorch_optimizer.neuron_mean` -> `pytorch_optimizer.optimizers.nero.neuron_mean`.
+
+### Docs
+
+* Add more visualizations. (#297)
+
+### Bug
+
+* Add optimizer parameter to `PolyScheduler` constructor. (#295)
+
+### Contributions
+
+thanks to @tanganke

docs/index.md

@@ -355,6 +355,10 @@ With the default bias-correction, Adam may actually make larger than requested g
 
 Correcting the norm of a gradient in each iteration based on the adaptive training history of gradient norm.
 
+### Cautious optimizer
+
+Updates only occur when the proposed update direction aligns with the current gradient.
+
 ## Frequently asked questions
 
 [here](docs/qa.md)

docs/visualization.md

@@ -42,6 +42,10 @@
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rastrigin_AdaMax.png)
 
+### AdamG
+
+![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rastrigin_AdamG.png)
+
 ### AdaMod
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rastrigin_AdaMod.png)
@@ -74,6 +78,14 @@
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rastrigin_AdaSmooth.png)
 
+### AdEMAMix
+
+![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rastrigin_AdEMAMix.png)
+
+### ADOPT
+
+![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rastrigin_ADOPT.png)
+
 ### AggMo
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rastrigin_AggMo.png)
@@ -134,6 +146,10 @@
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rastrigin_Fromage.png)
 
+### FTRL
+
+![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rastrigin_FTRL.png)
+
 ### GaLore
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rastrigin_GaLore.png)
@@ -250,6 +266,10 @@
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rastrigin_SM3.png)
 
+### SOAP
+
+![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rastrigin_SOAP.png)
+
 ### SophiaH
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rastrigin_SophiaH.png)
@@ -316,6 +336,10 @@
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rosenbrock_AdaMax.png)
 
+### AdamG
+
+![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rosenbrock_AdamG.png)
+
 ### AdaMod
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rosenbrock_AdaMod.png)
@@ -348,6 +372,14 @@
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rosenbrock_AdaSmooth.png)
 
+### AdEMAMix
+
+![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rosenbrock_AdEMAMix.png)
+
+### ADOPT
+
+![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rosenbrock_ADOPT.png)
+
 ### AggMo
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rosenbrock_AggMo.png)
@@ -408,6 +440,10 @@
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rosenbrock_Fromage.png)
 
+### FTRL
+
+![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rosenbrock_FTRL.png)
+
 ### GaLore
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rosenbrock_GaLore.png)
@@ -524,6 +560,10 @@
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rosenbrock_SM3.png)
 
+### SOAP
+
+![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rosenbrock_SOAP.png)
+
 ### SophiaH
 
 ![image](https://raw.githubusercontent.com/kozistr/pytorch_optimizer/main/docs/visualizations/rosenbrock_SophiaH.png)