Gradient Clipping

[anais2390]

Hi everyone,

Could you tell the correct way to perform gradient clipping with monai and pytorch ignite ?
Thanks a lot !

[Nic-Ma]

Hi @anais2390 ,

MONAI provides rich components for PyTorch training program, so actually, I think you can do gradient clipping just like a regular PyTorch program.
And if you are using ignite, I think @vfdev-5 may help provide some examples?

Thanks in advance.

[vfdev-5]

@anais2390 I think you can add a grad_clip_handler handler to IterationEvents.BACKWARD_COMPLETED event (

MONAI/monai/engines/trainer.py

Line 212 in 4b66746

engine.fire_event(IterationEvents.BACKWARD_COMPLETED)

)

from torch.nn.utils import clip_grad_norm_
from monai.engines.trainer import SupervisedTrainer, IterationEvents

trainer = SupervisedTrainer(...)

gradient_clip = 0.1

@trainer.on(IterationEvents.BACKWARD_COMPLETED)
def grad_clip_handler():
    clip_grad_norm_(self.network.parameters(), gradient_clip)

@Nic-Ma what do you think ?

[Nic-Ma]

Yeah, that’s also good and may be easier.

Thanks @vfdev-5

[anais2390]

Hi @Nic-Ma and @vfdev-5,

Thanks a lot for your help !
I had actually looked at the method through the IterationEvents, but I thought that the gradient clipping should be done before the backpropagation. Are you sure it is ok to do it on the event BACKWARD_COMPLETED ?

[vfdev-5]

@anais2390 please check the code:

MONAI/monai/engines/trainer.py

Line 212 in 4b66746

engine.fire_event(IterationEvents.BACKWARD_COMPLETED)

and you will see that IterationEvents.BACKWARD_COMPLETED is triggered after computing gradients, after loss.backward() but before applying them to the parameters: optimizer.step().

Here is another example in pytorch docs how to apply grad clipping with AMP. You can ignore AMP part and see that the steps are

• loss.backward
• grad clipping
• optimizer.step

Let me know if I misunderstood your question. Thanks

[anais2390]

This is very clear, thanks a lot for your help !