gradient checkpointing runs during validations in the training examples

[bghira]

Describe the bug

The gradient checkpointing is enabled via self.training, but then the log_validations also unnecessarily encounter this codepath.

I found that I have to disable this when running even under torch.no_grad(), looked and saw that the official examples do not do this either.

This gives a substantial performance boost more similar to a normal inference script running outside of a training loop.

Reproduction

Add print statements to the checkpointing function.

Logs

No response

System Info

Who can help?

@linoytsaban @yiyixuxu

[bghira]

this impacted every model I looked at while implementing layer-skipping for checkpointing.

[bghira]

@a-r-r-o-w see this one as well

[yiyixuxu]

we have this PR recently #9878
I think it is gradient checkpoint won't be anbled if you have torch.no_grads() now

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.