Why are optimizers set up inside Modules rather than in Trainers?

[augustebaum]

I hesitated to post this in the issue board with the design tag.

It feels strange to attach an optimizer to a Module and, e.g. pass the learning rate to the Module at initialisation, rather than pass the desired optimizer to the Trainer at training time. Can anyone explain the current approach, or direct me to some other resources?

The reason I ask is because I am currently migrating my research project on auto-encoders from pythae to lightning and in pythae the separation seems more logical to me.

[tshu-w]

IMHO, the optimizer is related to the model architecture, e.g. transformers and non-transformers architectures may require different optimizers.

[augustebaum]

Hey, thanks for your answer! I don't have much experience with Transformers so I don't really get the point. Nevertheless I would offer the counterpoint that it might be desirable to parametrize the learning process to compare two optimizers, or more rarely, to start learning with one optimizer and change it later. At the very least I would expect the docs to showcase an example of setting the optimizer from outside the Module instantiation, like how it would work if the Module __init__ took a optimizer argument.