overfit_batches duplicates entire train DataLoader causing out of memory

[jonathanking]

I would appreciate some help understanding the overfit_batches argument for the trainer.

When overfitting batches, it seems that the entire train dataloader is deepcopied (code in v1.5.10). In my case, this immediately results in my machine running out of RAM because of the size of this dataset. I also have many validation dataloaders, so I believe this compounds the issue.

Will this behavior of copying the entire dataloader be removed in a future release? I believe some other mechanism of duplication is necessary to avoid copying the data that is not included in the batches that are being overfitted.

I was trying to read #10877 to understand how the behavior would change, but I am unsure.

[rohitgr7]

hey @jonathanking !

the deepcopy for train dataloader to the validation dataloader has been removed from master and will be available in the next release soon.

[jonathanking]

Thank you very much!