Wrong time estimates in progress bar after epoch 0

[rubvber]

This issue seems to be happening ever since I manually updated PL to the current version on GitHub (because I wanted the new "strategies" module functionality). When I start training a model the time estimates in the progress bar are correct (time elapsed since beginning of epoch, time remaining, iterations/second). However, from epoch 1 (i.e. the 2nd epoch) onwards, it seems like it's not using the start of the current epoch any more to calculate its estimates, but instead keeps using the start of epoch 0. The time elapsed immediately starts out at a few minutes, and since the code thinks that it has completed only a few iterations in all that time, the speed (iterations/second) is wildly underestimated, as is the projected time remaining in the current epoch.

(Edit: It's maybe worth noting that the timer for the validation epochs does work correctly, starting at 0:00 every time.)

Has anyone else experienced this? Is this just a general bug or is it caused by a specific setting or usage? It's not a huge issue of course as everything still runs, but it's just annoying not to have this quality-of-life feature working properly.

[danthe3rd]

I have the same error. But only for training iterations - validation iterations correctly reset the timer it looks like

[rubvber]

Yes, I think it was the same for me. I have since reverted to the most recent "pip-release" and this resolved the issue for me so it does seem to be a bug in the newer version (or at least the version that was current on GitHub at the time of my original post).

[dtmoodie]

I observer this issue with version 1.6.2 from pip.