How to shuffle training data in every epoch?

[celsofranssa]

How could someone shuffle the training dataloader (using Datamodule) on each epoch?

[celsofranssa]

I think it is as simple as setting the shuffle parameter to True when returning the train_dataloader, correct?

As shown in the code snippet below:

class MyDataModule(pl.LightningDataModule):

    def __init__(self, params):
        super().__init__()
        self.params = params

    def setup(self, stage=None):

        # Assign train/val datasets for use in dataloaders
        if stage == 'fit' or stage is None:
            self.train_dataset = CodeSearchDataset(
                data_path=self.params.dir + "train.jsonl")

    def train_dataloader(self):
        return DataLoader(
            self.train_dataset,
            batch_size=self.params.batch_size,
            shuffle=True,
            drop_last=True,
            num_workers=self.params.num_workers
        )

[awaelchli]

You can set Trainer(reload_dataloaders_every_epoch=True) and if you have also shuffle=True in your dataloader, it will do that by creating a new dataloader every epoch.
That's my understanding.

[thomasahle]

This seems to now be called reload_dataloaders_every_n_epochs=1

[AnkushMalaker]

It seems to be the case that the default behavior is data is shuffled only once at the beginning of the training.
Every epoch after that takes in the same shuffled data. If we set reload_dataloaders_every_n_epochs=1, we get shuffling every epoch. In the docs located here, (in the video) William mentions that by default behavior is to shuffle every epoch. Maybe a mistake?

[rohitgr7]

from what I remember, the dataset is shuffled every time on each new iteration of dataloader when you set shuffle=True. reload_dataloaders_every_n_epochs will only be required only if you need to create a new dataset/dataloader after every n epochs.

[AnkushMalaker]

Thanks for the answer, @rohitgr7 .
To verify, I created a Dataloader for images, saved the first batch to disk. I can verify that the images in the first batch are different every epoch so you're correct.

Although, when using imgaug for augmentations, the augmentations applied to the first batch are always the same. That is, say the first image in first batch is rotated 45 degrees clockwise, the first image in first batch in next epoch also has the same augmentation (the image itself is different). Is this expected behavior, and how can I modify this?

[rohitgr7]

not sure here, you might need to look into imgaug documentation and see if there's any way to shuffle the applied augmentations.