When does loss.backward() get called when accumulating gradients?

[sachinruk]

Just wondering when backward gets called when you are accumulating over (say 8) batches. I had put a breakpoint in on_after_backward() and that seemed to be only getting called on the 8th iteration of training. According to this answer, in order to save on GPU memory, it seems its best to call loss.backward() on each iteration.

[tchaton]

Dear @sachinruk,

This behaviour was a bug and should have been resolved on master and on_after_backward should be called after each backward call. In the case of accumulating over (say 8) batches, you should see on_after_backward called 8 times on master, and only 1 time previously.

Best,
T.C

[sachinruk]

Is this merged into the latest lightning version, or still just in master? Thanks for the quick reply btw.

[sachinruk]

ok so checked in version 1.3.8 and it seems like it's still not fixed there. i.e. it waits accumulate_gradient times before on_after_backward is called.