-
-
Notifications
You must be signed in to change notification settings - Fork 39
Description
I think your ETA calculation is off by a factor of 100...
Through both my own manual calculations and visually inspecting the ETA while training, it seems that the decimal separator is placed in the wrong spot.
Example:
23-07-26 02:16:57.959 - INFO: <epoch: 33, iter: 167,900, lr:6.250e-06, t:31.6040s, td:0.0001s, eta:281.8024h> pix-l1: 4.2061e-04 fea-vgg19-l1: 7.0643e-01 l_g_gan: 1.4697e-02 l_d_real: 5.6621e-02 l_d_fake: 5.6456e-02 D_real: 4.0469e+01 D_fake: 3.7594e+01
Here we see that the ETA is 281.8024h(rs), when in fact it is much closer to 2.81hrs. This is consistent with every training run I've done so far, no matter how big or small the ETA actually is.
Even though I've gotten used to it now, I thought I might raise an issue here and let you know π