Metric names when using multiple dataloader for validation

[kantholtz]

Hi guys,

I am using multiple dataloaders for validation. This works great so far, but I have some questions regarding the logged metrics:

As far as I understand, lightning will automatically assign a metric suffix (/dataloader_idx_X). Is there any way for me to control that behaviour? Wandb assumes groups with that slash, and it means, that it groups by metric name and not validation set. However, there are scenarios where it makes much more sense to group by validation set, as they might look at completely different metrics. On the bottom, there is a screenshot on how it looks without any further wanbd configuration.

Apart from my question, this is more of a feature proposal: Can't we supply a (ordered) dictionary instead of a list via val_dataloader() in DataModules? So that it looks like this:

DataModule:

def val_dataloader(self) -> Union[Dict[str, DataLoader], List[Dataloader], DataLoader]:
   return {
      'scenario1': DataLoader('...'),
      'scenario2': DataLoader('...'),
  }

LightningModule

def validation_step(self, batch, batch_idx: int, loader: Union[str, int]):
   # do something
   self.log('my_metric', x)

Which then logs scenarioX/my_metric in case of the example or dataloader_idx_X/my_metric when a list is provided.

[tchaton]

Hey @kantholtz,

Thanks for sharing your ideas ! It shouldn't be too hard to add.
Let me see if I have some time for it this week.

Best regards,
T.C

[kantholtz]

If not, no stress: I wouldn't mind doing it myself and open up a PR :) But unfortunately I will only have time to do that from mid December on.

[tulay]

@kantholtz would you please mention PL version you use? When using multiple validation loaders, I am having issues with 1.0.8 most likely due to the line .

[kantholtz]

Sure, I am using 1.1.0-dev

[tulay]

Thanks. I pulled "pytorch-lightning==1.1.0rc1" and it runs fine.

[kantholtz]

@tchaton quick question: is there a single place, where I can simply remove the suffix without too much hassle or is it integrated more deeply? (My metrics have distinct names so there won't be any collision.)

[kantholtz]

For the curious if anyone needs a quick and dirty workaround: I set https://github.com/PyTorchLightning/pytorch-lightning/blob/7b7d4bb86d2c28ccc8b13953481f084d6f8accf7/pytorch_lightning/trainer/connectors/logger_connector/epoch_result_store.py#L152 to False and the suffix vanishes.

[marksibrahim]

I'm curious if there's a solution to this question? I came a across a similar unanswered question in the forums: https://forums.pytorchlightning.ai/t/name-of-dataloader-in-case-of-multiple-dataloaders/900

[marksibrahim]

Thanks for the suggestion, I was looking for an option to return a dictionary mapping a name to each loader from the val_dataloader or test_dataloader method for multiple sequential loaders (it seems Lightning only allows a list)?

Something akin to a sequential option for a dictionary:

class MyDataModule:

  def val_dataloaders(self):
    loaders = {"loader_1": loader, "loader_2": loader}
    return CombinedLoader(loaders, "sequential")

[rohitgr7]

CombinedLoader behaves a little differently. It doesn't iterate over your dataloaders independently but rather combines the batches into a single entity (dict) and sends it as a batch.
ref: https://pytorch-lightning.readthedocs.io/en/latest/guides/data.html#multiple-validation-test-predict-dataloaders

so if your dataloaders have different length, this won't work.

[marksibrahim]

I'm sorry my question wasn't clear—I understand CombinedLoader isn't sequential. What I'm wondering is whether there's an option to return multiple dataloaders sequentially with names (instead of a list):

class MyDataModule:

  def val_dataloaders(self):
    loaders = {"loader_1": loader, "loader_2": loader}
    # somehow specify a sequential option with the names above
    return loaders

[rohitgr7]

no there is no way for it right now, but you can use a mapping for it as a workaround!

def val_dataloaders(self):
     self.val_dl_map = {0: 'loader_1', 1: 'loader_2'}
     return [...]

def validation_step(self, batch, batch_idx, dataloader_idx):
    curr_val_dl_name = self.val_dl_map[dataloader_idx)
    ....

if you think returning dictionary should be supported, feel free to open an issue with a proposal :)

[surya-narayanan]

it seems now that one can't return a dictionary of loaders in the validation and test step- is that true?

[Zakhar17]

For those searching for the solution to the same issue, in the new version of lightning (I have 2.0.6), you are able to prevent dataloader_idx suffix using self.log(..., add_dataloader_idx=False) when logging inside LightningModule.