Multiple Validation Sets

[Jose-Bastos]

Hello, I'm trying to validate my model on multiple subsets of the initial validation set to compare performance. Reading this page I got the idea that returning a list contaning the multiple Dataloaders would be enough. My val_dataloader method became the following:

But this isn't working properly. I get the following error: "TypeError: validation_step() takes 3 positional arguments but 4 were given"
(it worked properly when I only used 1 validation Dataloader).
What am I doing wrong? Can someone help me with this? Or just point me to some more documentation on this.

Thanks in advance!

[ananthsub]

Could you share what your validation step implementation is?

[rohitgr7]

you must be missing the additional dataloader_idx required in the validation_step for multiple dataloaders
docs: https://pytorch-lightning.readthedocs.io/en/latest/guides/data.html#multiple-validation-test-predict-dataloaders

[rohitgr7]

@mariomeissner like this:

class LitModel(LightningModule):
    ...
    
    def validation_step(self, batch, batch_idx, dataloader_idx):
        ...
        if dataloader_idx == 0:
            self.log('val/acc', acc, add_dataloader_idx=False)
        elif dataloader_idx == 1:
            self.log('val/acc_secondary', acc, add_dataloader_idx=False)

[mariomeissner]

@rohitgr7 That makes a lot of sense. I always create new Accuracy instances for different metrics, but here we use the same acc for both validation datasets. Is that safe?

[mariomeissner]

I think it will interfere. For example, I currently calculate val_acc_best like this:

acc = self.val_acc.compute()
self.val_acc_best.update(acc)
self.log("val/acc_best", self.val_acc_best.compute(), on_epoch=True, prog_bar=True)

But this acc is mixing both validation datasets now, returning the wrong value, right?

I guess I have to keep separate metrics for each validation set...

[rohitgr7]

yes.. you should create a different metric instance for each dataloader as well if you are using Modular metrics. They contain some internal states so mixing up results for different datasets will result in incorrect computation of the final metric.

[mariomeissner]

Thanks again for your help!