How to load the parameters of optimizer and schedulers dynamically?

[marsggbo]

My experiment setting is like this:

1. In the first 10 epochs, I need to use trainer1 to train model1
2. In the second 10 epochs, I need to instantiate two pairs of (trainer2-1, model2-1) and (trainer2-2,model2-2). Both need to inherit the model/optimizer/scheduler weights from trainer1 and model1, and are trained separately.
3. In the third 10 epochs, similarly, we create four pairs of trainers and models

model.load_from_checkpoint or model.load_state_dict only load model weights, but optimizer and scheduler's parameters are ignored. I wonder how to load the parameters of optimizers and schedulers dynamically?

[akihironitta]

You can utilise PyTorch Lightning's checkpoints as they include states of optimizer and learning rate schedulers, too:

trainer.save_checkpoint("example.ckpt")
checkpoint = torch.load("example.ckpt")

See the documentation page for details: https://pytorch-lightning.readthedocs.io/en/stable/common/checkpointing.html#checkpoint-contents