Gradient Checkpointing #11085
Answered
by
polm
brosand
asked this question in
Help: Other Questions
Gradient Checkpointing
#11085
-
|
Is there any way to enable gradient checkpointing for model development? |
Beta Was this translation helpful? Give feedback.
Answered by
polm
Jul 6, 2022
Replies: 1 comment
-
|
Do you mean saving training state after every N iterations or something? If so, no, that's not a feature - training just saves the best and last models. This is partly related to resuming training being kind of difficult at the moment. If you wanted to save models more frequently you should be able to do so by modifying the training loop code pretty easily. |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
polm
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Do you mean saving training state after every N iterations or something? If so, no, that's not a feature - training just saves the best and last models. This is partly related to resuming training being kind of difficult at the moment.
If you wanted to save models more frequently you should be able to do so by modifying the training loop code pretty easily.