Data augmentation and reload_dataloaders_every_epoch

[dianemarquette]

Hi!

I'm training my neural network with Pytorch Lightning and MONAI (a PyTorch-based framework for deep learning in healthcare imaging). Because my training dataset is small, I need to perform data augmentation using random transforms.

Context

I use MONAI's CacheDataset (basically, a PyTorch Dataset with cache mechanism). From what I understood, , CacheDataset will cache the consistent result of the transforms until the first random transform, then reuse the cache content and apply the remaining random transforms (such as Gaussian noise, random intensity shift, etc.) for every epoch. As a result, the dataloader trains the network with a different dataset at each epoch.

In the video presenting the reload_dataloaders_every_epoch flag, William Falcon mentions that:

By default, Lighthning only loads your dataset once (so that you don't occur the cost of downloading that data and process it every single time). On every epoch, Lightning shuffles the data and feeds it into the training loop.

My questions

Did Lightning add a cache mechanism to load the data once? Must I use the reload_dataloader_every_epoch flag to do data augmentation or else my random transforms will only be applied once (therefore defeating my data augmentation goal)?

Thanks in advance for your explanation :)

[carmocca]

Did Lightning add a cache mechanism to load the data once?

No, the flag just means that we call LightningModule.train_dataloader() every epoch if enabled, thus creating a new DataLoader instance.

Must I use the reload_dataloader_every_epoch flag to do data augmentation or else my random transforms will only be applied once

If I understand correctly, no.
The transformations are applied directly in the Dataset, so every time an item is consumed from it, the random transforms should be applied regardless of whether the DataLoader has or hasn't been recreated.

[dianemarquette]

@carmocca Thanks for the quick response. It makes sense now! Happy to learn that I don't need to enable this flag and occur the extra cost of creating a new DataLoader instance to perform data augmentation :)