How to profile the processes of backward, optimizer.step, all_reduce?

[marsggbo]

We can easily count the time of a forward, but how can we count the time of processes of loss.backward and optimizer.step?

[awaelchli]

@marsggbo Is the profiler not already giving you this information?
https://pytorch-lightning.readthedocs.io/en/latest/advanced/profiler.html