Distributed Training in PyG: How to partition the data that does not fit in memory?

[s-udhaya]

Hi,
Thank you for this excellent library! I'm working with a large graph dataset that exceeds available memory. While the distributed training documentation (https://pytorch-geometric.readthedocs.io/en/latest/tutorial/distributed_pyg.html) is helpful, I have a question about dataset partitioning:

Since the documentation indicates partitioning occurs on in-memory data, how can I effectively leverage the distributed training system when my entire dataset doesn't fit into memory? Are there recommended strategies or examples for this scenario?

[Amyr14]

Hey. I'm not a developer of pyg, but I submitted a very similar question a while ago. You should take a look on sampling techniques. Pyg has some implemented sampling interfaces in the sampler module. If none of them are exactly what you need, you can always implement it yourself using the BaseSampler abstract class. Hope this helps.

[s-udhaya]

Hey, thanks for sharing the links. I will look into that.

[Amyr14]

Forgot to tell you that you can also use a data loader, since they implement a sampler and wrap the sampler output in a iterator that you can use to perform mini-batch training. If you still need to customize the sampling routine, you can use the NodeLoader and/or LinkLoader class and implement a BaseSampler for it.
Also, in the link you provided there is a section explaining how to use distributed data loaders.