What does model checkpoint do under the hood?

[FeryET]

Hi.

I'm trying to implement a GAN loop, and after restoring model from a checkpoint, I'm seeing a complete discripency between before checkpoint losses and after checkpoint losses. I'm suspecting two things.

1. I freeze and unfreeze my generator and discriminator in the training loop, Maybe lightning does not save optimizer states for frozen layers? If so, I should unfreeze the whole model at the end of the training epoch.
2. I also suspect that the optimizer parameters are not saved for further training, if that's true, how can I completely restore the model state to further train the model.

[rohitgr7]

1. I freeze and unfreeze my generator and discriminator in the training loop, Maybe lightning does not save optimizer states for frozen layers? If so, I should unfreeze the whole model at the end of the training epoch.

possibly yes, since it relies on optimizer.state_dict() to retrieve the states and I believe if any layer is freezer then optimizer doesn't track the state for it and thus it's saved at all.

2. I also suspect that the optimizer parameters are not saved for further training, if that's true, how can I completely restore the model state to further train the model.

optimizer states are also saved: https://pytorch-lightning.readthedocs.io/en/latest/common/checkpointing.html#checkpoint-contents