Log All Hyperparameters/Arguments

[AlessandroW]

Hey, is there a best practice for logging all arguments passed to the ArgumentParser?

Due to using the DataModule, the data related arguments are not tracked by the LightningModule

My goal is to:
a) log all parameters in the hparams.yaml
b) log a subset in TensorBoard.

My "solution" is:

from argparse import ArgumentParser

import pytorch_lightning as pl

class MyModel(pl.LightningModule):

    @classmethod
    def from_argparse_args(cls, args, **kwargs):
        result = pl.utilities.argparse.from_argparse_args(cls, args, **kwargs)
        #HACK: Log all Hyperparameters
        params = vars(args)
        params.update(**kwargs)
        result._hparams_initial = params
        return result
  
 ...
parser = ArgumentParser()
parser = MyTransforms.add_model_specific_args(parser)
parser = MyDataModule.add_model_specific_args(parser)
parser = pl.Trainer.add_argparse_args(parser)

parser = MyModel.add_model_specific_args(parser)
args = parser.parse_args()

transforms = MyTransforms.from_argparse_args(args)
data = MyDataModule.from_argparse_args(args, transforms=transforms)

model = MyModel.from_argparse_args(args)

trainer = pl.Trainer.from_argparse_args(args)
trainer.fit(model, data)

Then, in Trainer.pre_dispatch() all hyperparameters will be logged.
This means all hyperparameters will be logged in TensorFlow but that is okay for now.

Does a better way exist already?

[awaelchli]

Outside the LightningModule I believe you should be able to just call logger.log_hyperparams(**vars(args)) with your argparse arguments saved in the local variable args.

[AlessandroW]

Thanks for replying!

Yes, that would work generally. However, in the TensorBoard case I need to include the metrics, so that they appear in hparams.
Otherwise, metrics is None and the default hp_metric = -1 will be used (or nothing will be logged at all). https://github.com/PyTorchLightning/pytorch-lightning/blob/d6470bf1937e51e037a7f94a55ad76898e5ae103/pytorch_lightning/loggers/tensorboard.py#L178
AFAIK this behavior was implemented due to #1228

So, as explained in logging-hyperparameters, you need to call

def on_train_start(self):
        self.logger.log_hyperparams(self._hparams_initial, {"hp/metric_1": 0, "hp/metric_2": 0})

to log all hyperparameters and the hyperparameter metrics.

When I call log_hyperparams directly it logs the hyperparams to the .yaml but not in TensorBoard.

logger = TensorBoardLogger(save_dir="/tmp", default_hp_metric=False)
logger.log_hyperparams(params=vars(args), metrics={"hp/metric_1": 0, "hp/metric_2": 0})

This only works when calling the logger in on_train_start.

Can you explain to me where the metrics are updated, as this is the only call to log_hyperparams?
Then I'd might be able to debug this further.

[awaelchli]

I found this conversation here, in particular this answer: #2974 (comment)

I remember this is a limitation/design of TB, we can't call the method multiple times. So maybe the answer to your original question

Does a better way exist already?

is no at the moment with this setup .(

Related: A feature request for hyper parameters in datamodule is here:
#3769

On another note, personally I found similar problems with argparse and instantiating multiple components using the same parser is not very flexible. I prefer to instantiate the model with kwargs named parameters instead of using the from_argparse_args, and for more sophisticated configuration a third party library like Hydra.

Recently Lightning added also it's own lightweight LightningCLI, it's in beta and will be released in 1.3.