Iterate over all validation data while using limit_val_batches

[SPlanzer]

I have 10,000s of images for training a semantic segmentation model.

I am therefore using limit_val_batches (along with val_check_interval) to limit the number of valuation batches per epoch

I however note (i.e Pytorch Lightning limit_val_batches and val_check_interval behavior - Stack Overflow) that when using limit_val_batches=N, the first N batches from the dataloader are returned for each validation iteration. Training therefore only every sees the first ‘N’ validation batches when using limit_val_batches and never the rest that makes up the bulk of validation examples

Rather than using the same ‘N’ batches starting at index 0 for each validation epoch, I would like the dataloader when using limit_val_batches to sequentially chunk through all the validation data (e.g 0-19 batches the first epoch and then 20-39 the next... and so on)

How would I go about implementing this behavior with limit_val_batches? or is this not the expected thing to do?

[tshu-w]

limit_val_batches is not the expected thing to do.

[SPlanzer]

Thanks @tshu-w.

In that case, what is the strategy when your validation dataset (or training set) has 10,000 plus images in it and you don't want validation to validate against all these images every epoch, but rather chunk through the dataset each epoch a small bit at a time in a sequential manner?

[tshu-w]

I have not encountered a similar need for the time being. But you may try to set reload_dataloaders_every_n_epochs to 1 in trainer and return part of the images in the val_dataloader like:

    def val_dataloader(self) -> EVAL_DATALOADERS:
        return DataLoader(
            dataset=self.data["val"][self.start:self.start+self.chunk_size],
            batch_size=self.hparams.batch_size,
            num_workers=self.hparams.num_workers,
            pin_memory=self.hparams.pin_memory,
            persistent_workers=self.hparams.num_workers > 0,
            shuffle=False,
        )
        self.start = self.start + self.chunk_size

[SPlanzer]

For anyone with a similar problem, aniketmaurya provided an answer at over at the now defunct pytorch forums which also solves this issue.

it is as follows:

Hey @aeolian, you can shuffle your validation dataloader so that for each epoch random images would be validated.

Also, we have recently migrated to GH Discussions for answering community questions. Feel free to ask there if you’re still having some issues.

If you are using this method and want to compare results across training you will want to use the worker_init_fn() to preserve reproducibility: