Why is trainer.init_optimizers called in so many places?

[daniellepintz]

I see we call trainer.init_optimizers in many places in the code. I am wondering why don't we just call it once, when we initialize the trainer, and then for the existing calls instead just access trainer.optimizers, trainer.lr_schedulers, trainer.optimizer_frequencies?

It is defined in the TrainerOptimizersMixin:
https://github.com/PyTorchLightning/pytorch-lightning/blob/c33df2639f19d49c5e7520294e3221efe402d684/pytorch_lightning/trainer/optimizers.py#L32

It is called in these places:
https://github.com/PyTorchLightning/pytorch-lightning/blob/c33df2639f19d49c5e7520294e3221efe402d684/pytorch_lightning/plugins/training_type/deepspeed.py#L458

https://github.com/PyTorchLightning/pytorch-lightning/blob/c33df2639f19d49c5e7520294e3221efe402d684/pytorch_lightning/plugins/training_type/training_type_plugin.py#L248

https://github.com/PyTorchLightning/pytorch-lightning/blob/c33df2639f19d49c5e7520294e3221efe402d684/pytorch_lightning/tuner/lr_finder.py#L109

https://github.com/PyTorchLightning/pytorch-lightning/blob/c33df2639f19d49c5e7520294e3221efe402d684/tests/helpers/pipelines.py#L87

[rohitgr7]

1 and 3 seems ok since it's just a sequential call.
last one is in tests so possibly maybe not a problem.
for 2nd one (deepspeed), maybe this can be avoided (but not 100% sure)
for lr_finder (tuner) this possibily needs some refactoring after v1.5 since sometimes it just doesn't work or gives unexpected results. So maybe at that time we might come up with a better solution for this.

[daniellepintz]

thanks @rohitgr7!! I am curious what others think here as well, @Borda @carmocca @tchaton

[carmocca]

yeah, ignore the tuner example, we would like to revamp it completely at some point.