Skip to content

Commit 33e606e

Browse files
authored
Update README.md
1 parent f6dcd54 commit 33e606e

File tree

1 file changed

+3
-4
lines changed

1 file changed

+3
-4
lines changed

README.md

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -23,9 +23,8 @@ This package implements the Newton type and Fisher type preconditioned SGD metho
2323
[5] https://medium.com/@yaroslavvb/optimizing-deeper-networks-with-kfac-in-pytorch-4004adcba1b0
2424

2525
#### Misc
26+
*Comparison on LeNet5*: We halve the step size every epoch. The Newton type method occasionally gets test classification error rate below 0.7%. Code reproducing KFAC's results are at ./misc/demo_LeNet5_KFAC.py.
27+
![alt text](https://github.com/lixilinx/psgd_torch/blob/master/misc/mnist_lenet5.jpg)
2628

27-
*Comparison on autoencoder*: I compared several methods on the autoencoder benchmark [5], and the results as shown as below. KFAC uses batch size 10000, while other methods use batch size 1000. KFAC's step size reduces to 0.1, otherwise, its test loss is too high. Second order methods converge faster and generalizes better. Actually, our Newton type method converges to test loss less than 10 even with code length 8!
29+
*Comparison on autoencoder*: I compared several methods on the autoencoder benchmark [5], and the results as shown as below. KFAC uses batch size 10000, while other methods use batch size 1000. KFAC's step size reduces to 0.1, otherwise, its test loss is too high.
2830
![alt text](https://github.com/lixilinx/psgd_torch/blob/master/misc/mnist_autoencoder.jpg)
29-
30-
*Comparison on LeNet5*: The Newton type method occasionally gets test classification error rate below 0.9%. Code reproducing KFAC's results are on https://github.com/lixilinx/psgd_torch/blob/master/misc/demo_LeNet5_KFAC.py
31-
![alt text](https://github.com/lixilinx/psgd_torch/blob/master/misc/mnist_lenet5.jpg)

0 commit comments

Comments
 (0)