Nested hparams?

[FeryET]

Hi.

I have a nested model inside my LightningModule which has a set of unique variables. The nested model can change (can be either LSTM/Transformer etc), but the overall PL blackbox should not change. What I want to do is to store the hyperparameters related to the nested model in hparams, but I want to make it hierarchical, and have nested hparams.

Something like this: self.hparams['net'] = {} # a dict. How can I achieve this? Currently if I call self.save_hyperparameters(net_config) the hparams dict will update its values with the net_config dictionary that I have and it will be unnecessarily crowded.

[Blaizzy]

Hi @FeryET

Prince Canuma here, a Data Scientist at Neptune.ai

I have a couple of questions and a potential answer to your issue.

You mentioned that your nested model can change (can be either LSTM/Transformer etc).

• Can you elaborate on this and give me an example of what you are trying to achieve?

You also talk about storing your hyperparameters related to the nested model in hparams variable but it had to be in a hierarchical/nested structure.

• How do you plan to use these hparams? Is it gonna be used on a downstream task?
• I ask this because the solution you are looking for will only persist in memory and will be erased after the script finishes running. Unless you only need it while your script is running. Otherwise, you would need an external place to store them.

Potential solution
Here is a code example of how you can save your nested params in hparams using your LightningModule:

class AutoArgsModel(LightningModule):
     def __init__(self, **params):
         super().__init__()
         # automatically assign arguments
         self.save_hyperparameters()
     def forward(self, *args, **kwargs):
         pass

model = AutoArgsModel(bs=1, lr={'start':0.05, 'end':0.1})

print(f'All params:\n{model.hparams}')
print('Access elements:', model.hparams['lr']['start'])

Output:

All params:
"bs": 1
"lr": {'start': 0.05, 'end': 0.1}

Access elements: 0.05

PTL Docs: https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html#save-hyperparameters

[FeryET]

Hi. I'm trying to use LightningModule as a wrapper for my other downstream tasks. I might have Transformer, LSTM, CNN or another different architecture altogether in the backbone while other parts of the wrapper stay the same, e.g. metrics, loss functions, optimizers, etc.

For this issue, I ultimately tried incorporating factory pattern, implementing a serializable config object for each of my backbones, and indicating them using LightningCLI config file parser, which is then tracked by MLFlow logger as artifacts. The model's weights are also registered via the experiment tracking system which makes reloading the model's weights really easy.

[Blaizzy]

Great, since you are using the hparams for a downstream task I was about to suggest you use an experiment tracker because it makes your life easier in terms of storing and retriveing them.

I see you went the long route by incorporating a factory method pattern. Can you give a full example of the different stages (preferably with some snippets and outputs)?

I want to understand your use case, especially why you serialize your config object for each backbone. I also want to understand how you are using it the CLI and tracking your experiment metadata. Perhaps there is room for improvement 🤓 and I would be more than happy to help you with any ideas you and/or I might have from this exercise.

[FeryET]

Great, since you are using the hparams for a downstream task I was about to suggest you use an experiment tracker because it makes your life easier in terms of storing and retriveing them.

I see you went the long route by incorporating a factory method pattern. Can you give a full example of the different stages (preferably with some snippets and outputs)?

I want to understand your use case, especially why you serialize your config object for each backbone. I also want to understand how you are using it the CLI and tracking your experiment metadata. Perhaps there is room for improvement nerd_face and I would be more than happy to help you with any ideas you and/or I might have from this exercise.

I'm currently in the middle of my exams and cannot give you snippets but will do in a few weeks ;).

For the case of serializing, it's because I'm trying to scale up the model to fast prototyping for hyperparameter tuning experimentation, and tracking configs and reproducing them is essential. Currently LightningCLI uses a jsonargparse backend which can initialize my config files easily, and those config files are also easily saved in my checkpoint, linking them together.

[Blaizzy]

No problem, best of luck with the exams!

Let's chat more once your exams are done. 😃

[Blaizzy]

Hey @FeryET