Question: How to switch models ?

[yukkyo]

Thank you for the wonderful library! 😄
I looked at the example below and didn't know how to compare the models if the datasets were the same.

https://github.com/williamFalcon/pytorch-lightning/blob/master/examples/new_project_templates/lightning_module_template.py

There are several ways to do this.

1. Switch by __build_model () function in "class LightningTemplateModel (LightningModule)"
2. Pass your Model to __init __ () of "class LightningTemplateModel (LightningModule)"
3. Create a Model Class that inherits "class LightningTemplateModel (LightningModule)"

Which is a good practice?
Or do you know a better way?

[williamFalcon]

@yukkyo thanks! i'm not quite understanding what you want to do.

The normal flow is:

1. Create a subclass of LightningModule.
2. define a system there (could be multiple networks, etc...)
3. Pass into trainer.

[yukkyo]

@williamFalcon
Thank you for your quick reply! :)

First of all, I don't want to write the same process every model (training_step(), validation_step(), tng_dataloader(), and so on).

As you say, if I want to compare multiple models (and training_step() etc. are the same),

1. Add params (like model_type) to self.hparams byadd_model_specific_args()
2. Create a function in LightningModule that creates a different network depending on the model_type
3. When initializing LightningModule, create a network according to model_type and start training

Is it correct?

[williamFalcon]

oh, you have different systems?
Create a LightningModule for each one. You can use the same trainer for both in a different main script.

File 1:
Model1(LightningModule).

File 2:
Model2(LightningModule).

File 3:

if want_model_1:
    model = Model1()
else:
    model = Model2()

trainer = Trainer(...)  
trainer.fit(model)

[yukkyo]

@williamFalcon
Does that mean I have to define a LightningModule for each network and define exactly the same function in the Lightning Module (ex. Training_step())?

[williamFalcon]

i’m sorry, i’m not sure i understand.

If the training_step is the same then define 1 Module.

what are you trying to do? can you write pseudocode?

[yukkyo]

Sorry for not being able to speak well.

In my case, I want to compare metrics of Network1 and Network2.
I came up with three ways but I don't know which one is better.

Method1: Define LightningModule for each Network

class LightningTemplateModel1(LightningModule):
    """
    Sample model to show how to define a template
    """
    def __init__(self, hparams):
        # build model
        self.__build_model1()

    def __build_model(self):
	# Define of Network1
        # Long code

    def forward(self, x):
        # Long code
        return logits

    def training_step(self, data_batch, batch_i):
        # Long code

    def validation_step(self, data_batch, batch_i, dataloader_i):
        # Long code


class LightningTemplateModel2(LightningModule):
    """
    Sample model to show how to define a template
    """
    def __init__(self, hparams):
        # build model
        self.__build_model2()

    def __build_model(self):
        # Define of Network2
        # Long code

    def forward(self, x):
        # Long code
        return logits

    def training_step(self, data_batch, batch_i):
        # Long code

    def validation_step(self, data_batch, batch_i, dataloader_i):
        # Long code

Method2: Switch Networks by hparams

class LightningTemplateModel(LightningModule):
    def __init__(self, hparams):
        # Switch model by hparams
        if hparams.model_type == 'Network1':
            self.__my_model = build_model1()
        else:
            self.__my_model = build_model2()

    def forward(self, x):
        return self.__mymodel(x)

    def training_step(self, data_batch, batch_i):
        # Long code
        return

    def validation_step(self, data_batch, batch_i, dataloader_i):
        # Long code
        return

    @staticmethod
    def add_model_specific_args(parent_parser, root_dir):  # pragma: no cover
        parser = HyperOptArgumentParser(strategy=parent_parser.strategy, parents=[parent_parser])
        parser.add_argument('--model_type', default='Network1', type=str)
        return parser

Method3: Use subclass

class MyLightningTemplateModel(LightningModule):
    def __init__(self, hparams):
        self.__build_model()

    def __build_model(self):
        raise NotImplementedError()

    def forward(self, x):
        raise NotImplementedError()

    def training_step(self, data_batch, batch_i):
        # Long code
        return

    def validation_step(self, data_batch, batch_i, dataloader_i):
        # Long code
        return

class LightningTemplateModel1(MyLightningTemplateModel):
    def __init__(self, hparams):
        self.__build_model()

    def __build_model(self):
        # Define of Network1
        # Long code
        return

    def forward(self, x):
        # Long code

class LightningTemplateModel2(MyLightningTemplateModel):
    def __init__(self, hparams):
        self.__build_model()

    def __build_model(self):
        # Define of Network2
        # Long code
        return

    def forward(self, x):
        # Long code

[williamFalcon]

yeah, method1 is the intended pattern.

[yukkyo]

Thank you! I'll do my best to write!

[williamFalcon]

wait actually. if model1 is just a network, and model2 is a different architecture then use method2.

[yukkyo]

I assume that Network1 is YOLO and Network2 is RetinaNet.

[williamFalcon]

yeah. just use hparams to toggle.

the lightningModule is meant for a full system. you are just specifying a particular architecture in a broader system

[yukkyo]

Thank you! That was a really big help!

[edwardpwtsoi]

@williamFalcon
May I know when would you suggest the method 3 being used? I have a similar situation while my modules share some custom methods but not the training step nor validation step. I wonder would that be the case method 3 should be used? Thank you for your attention!

[tchaton]

Dear @edwardpwtsoi,

I would advice for the solution which takes the less line of code and maximize readability. Solution 3 seems a good approach, but you would need to be careful when modifying parent class to not break children.

Best,
T.C

[Gladiator07]

I have one more question. Let's say we have two models, Model A and Model B, and we can toggle them by passing the model_type in LightningModule. But, what if both of the models accept different arguments, I can't keep changing the arguments of the LightningModule. I understand that we can pass kwargs for model specific parameters, but I like to see actually what parameters are being passed, also the loss function may be different and it accepts another set of parameters which are different for different losses. I attached a pseudo-code below to show my current approach. Is there any better way to do this?

class LightningTemplateModule(pl.LightningModule):
    def __init__(self, model_type: str, model_specific_params : dict, loss_specific_params: dict)
    # long code
    if model_type == "modelA":
        self.model = ModelA(**model_specific_params)
       # other if-else statements
    
    if loss_type == "lossA":
        self.loss_fct = LossA(**loss_specific_params)
       # other if-else statements

    # training_step, validation_step, etc, etc
    
    @staticmethod
    def from_configuration(cfg : Dict[str, Any]):
        return LightningTemplateModule(model_type=cfg['model_type'],
                                       loss_type = cfg['loss_type'],
                                       model_specific_params = cfg['model_params'],
                                       loss_specific_params = cfg['loss_params'])
# configuration loaded from a yaml file
cfg = load_yaml(config_path)

model = LightningTemplateModule.from_configuration(cfg)