Where should I place my optimizer.zero_grad?

[vence-andersen]

Hie,

After watching your course content now, I am reading others code to see how other solve a problem. The one thing I noticed was, some had the "optimizer.zero_grad()" at the last like they had "loss.backward()", "optimizer.step()", and then finally they had "optimizer.zero_grad()". From your unofficial PyTorch song (I just sing it every time to remember the steps) and of course your course contents I thought we need to follow an order. Can I interchange the position of all three or like I can only move the "zero_grad" to the last. Could you please clarify it.

Thanks in advance.

[abhishekmann]

Hi,

Thing is you can place the optimizer.zero_grad() anywhere as long as the gradients are emptied before the next optimization step. So placing it at the beginning right after when you're iterating over dataloaders or right at the end right after optimizer.step() or somewhere in between, it all works. Just make sure the gradients are empty before optimizer.step() is called that is the keypoint.

Hope that helps.