Neptune logs folder is flooded when using track_grad_norm

[alessiobonfiglio]

🐛 Bug

If you try to log the norms of your gradients during training, they will be saved in the same folder as all the other logs, flooding that folder with a lot of attributes if you are training a model with a lot of weights

To Reproduce

neptune_logger = NeptuneLogger(...)
trainer = pl.Trainer(logger=neptune_logger, track_grad_norm=2, ...)

Expected behavior

I'm not sure what is the best solution to this problem, but I would suggest to put these logs in another folder inside "logs", like "logs/grad_norms" (it can be done as shown here under "namespaces")

Environment

• PyTorch Version: 1.2.10
• OS: Linux

[awaelchli]

I'm not familiar with Neptune so much. Other loggers have grouping, i.e., if the metric is named grad_norm/xyz then it will group all under a grad_norm section.
Is it similar in Neptune? Are you able to help fix this?

[alessiobonfiglio]

It's like that also for neptune (as shown in the link I put above), but, right now, lightning logs gradients like this grad_2.0_norm_decoder.blocks.0.conv1.0.weight, while we would prefer something more like this grad_2.0_norm/decoder.blocks.0.conv1.0.weight (or even like this grad_2.0_norm/decoder/blocks/0/conv1/0/weight)

If this is the default behavior we want then we can trivially change line 47 in pytorch_lightning/utilities/grads.py from
norms[f'grad_{norm_type}_norm_{name}'] = round(param_norm, 4)
to
norms[f'grad_{norm_type}_norm/{name}'] = round(param_norm, 4)
(for the second solution we just have to replace "." with "/" in name, still trivial)

PS: I don't know so well lightning's codebase, my fixes assume that grad_norm() in pytorch_lightning/utilities/grads.py is used only for tracking the gradient norms, but maybe the current format is necessary somewhere else

[awaelchli]

Yes that could be a possibility.
Want to try the change and send a PR?

[alessiobonfiglio]

I could if you want, just tell me which change from the two I suggested

[awaelchli]

The first option with just one forward slash at the top level should be fine. We just need to make sure all the information is still displayed in TensorBoard, which is the default logger.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!