How to optimally load big dataset

[duongnlt]

My dataset contains ~5M pairs of graph and can not be loaded fully using InMemoryDataset. So my initial solution is storing each pair in a .pt file and loading them when training (using Dataset).
The problem with my approach is it make the training too slow. Does anyone have a better solution?

[wsad1]

Try increasing num_workers argument in your DataLoader, with this you could load the next batch while the current batch is being processed. This will help provided dataloading is the bottlenect.

[duongnlt]

I tried to increase the num_workers to 12 but the speed did not improve much.

[rusty1s]

We currently don't have a better solution to this integrated in PyG. I see people for example using databases to store their data and retrieve it. Alternatively, one approach one can use is storing graphs in batches, which would result in faster I/O.

[duongnlt]

Thank you, storing graphs in batches seems to be the solution here.