Override hyparameters when continuning to train from checkpoints

[LeCongThuong]

I can not find any way to override hyparameters when continuing to train with pl.Trainer(resume_from_checkpoint='path_to_checkpoint')

In function configure_optimizers(self) in my model, I setup variable head_lr_scheduler

def configure_optimizers(self):
        head_optim = torch.optim.Adam(self.head.parameters(), lr=3e-4)
        head_lr_scheduler = torch.optim.lr_scheduler.MultiStepLR(head_optim, milestones = [20, 40, 81, 120], gamma=0.5)
        return [head_optim], [head_lr_scheduler]

When continuing to train, I want to change milestones from [20, 40, 81, 120] to [20, 40, 81, 120, 140, 180] but model does not change. I realized that by using lr_monitor callbacks

In main function:

trainer = pl.Trainer(
        resume_from_checkpoint='path/to/model',
        **cfg.trainer,
        logger=tensorboard,
        callbacks=callbacks,
    ) 
model_module = model_module.load_model(checkpoint_path='path/to/model')
trainer.fit(model_module, train_dataloader=train_loader)

Any help!!!

[awaelchli]

Hi
resume_from_checkpoint will restore the trainer state completely from the checkpoint. It is meant to resume exactly from the previous state it was in (including state of optimizer, scheduler etc.). There is a feature request for the ability to control what to restore.

Can you try this?
model = YourModelClass.load_from_checkpoint("path/to/ckpt", milestones=[20, 40, 81, 120, 140, 180])

(make milestones a model attribute and reference it in the configure_optimizers hook).