How to do model comparison with pytorch lightning

[ajayrfhp]

I want to test performance of my model on a few different feature sets and visualize losses of different features on the same plot in tensorboard.
My current work flow is,

class LightningWrapper(pl.LightningModule):
    def__init__(self, features):
        self.features = features 

    def training_step(self, batch, batch_idx):
        ....
        .....
        self.logger.experiment.add_scalars("version_0", { f"{self.features}" : loss})

trainer = pl.Trainer()
for feature_set in potential_feature_sets:
    model = LightningWrapper(feature_set)
    trainer.fit(model)

But this is creating different versions inside tensorboard logging. Does each call to Lightning module create a new version ? Is this because the loggers are not shared across feature sets in my current design ? How do I share logger across different feature sets or different models ?

[ajayrfhp]

Found a rather simple solution that worked for me. Put different features/different models under different directories and fire up tensorboard in root directory.

class LightningWrapper(pl.LightningModule):
    def__init__(self, features):
        self.features = features 

    def training_step(self, batch, batch_idx):
        ....
        .....
        self.logger.experiment.add_scalar("loss", loss)


for feature_set in potential_feature_sets:
    logger = pl.loggers.TensorBoardLogger(f"lightning_logs/{num_features}", version="0")
    trainer = pl.Trainer(logger=logger)
    model = LightningWrapper(feature_set)
    trainer.fit(model)