How to log parameters' norms and parameters' gradients' norms during .fit()?

[RylanSchaeffer]

I have a two related questions regarding logging parameters' norms and parameters' gradients' norms.

1. The Trainer class has a track_grad_norm flag. I don't know what "tracking" means. Does this mean that parameters' norms are logged?
2. Relatedly, how do I log the norms of my parameters during .fit()?

[aniketmaurya]

Hi @RylanSchaeffer, to log the parameter norms or grad norm you can do something like - grads = {n:p.grad.cpu() for n, p in model.named_parameters()} and then calculate the norm.

To log the norms of your parameters during .fit(), you can log it in any of the available hooks like training_step, on_train_epoch_end or so.

[aniketmaurya]

Also, check out this tutorial on the track_grad_norm explanation.

[rohitgr7]

it's mentioned over here: https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html?highlight=track_grad_norm#track-grad-norm