Big dataset, what is the best way to proceed?

[TommasoBendinelli]

Hello,
I am trying to train a model with a big dataset (Over 40 GB). This dataset is too big for being loaded at the very beginning on the RAM. So I was planning to load it into chunks. However, with the current dataloader API only way of workings are clear to me

1. Load the entire dataset at the very beginning before training (i.e. in the init in the dataloader)
2. Load on a sample at the time during the getitem phase.

Ideally, I would need a hybrid solution where I am loading a chuck of data samples, a process entirely, and then move to the next chunks (with multiple workers working in parallel). Is it possible, if so, how?

Best,
Tommaso

[ananthsub]

Have you tried using an iterable dataset? https://pytorch.org/docs/stable/data.html#iterable-style-datasets

[awaelchli]

I think what you are asking for is exactly what PyTorch Dataloaders already implement. They load chunks (batches) with multiple async workers in parallel. There is always never the need to load the whole dataset to RAM. If you want one sample at a time, you set batch_size=1 (default)

[superhero-7]

Actually, I wann do the same things, how do you fix this problem finally?