Distributed Training in PyG: How to partition the data that does not fit in memory?

[s-udhaya]

Hi,
Thank you for this excellent library! I'm working with a large graph dataset that exceeds available memory. While the distributed training documentation (https://pytorch-geometric.readthedocs.io/en/latest/tutorial/distributed_pyg.html) is helpful, I have a question about dataset partitioning:

Since the documentation indicates partitioning occurs on in-memory data, how can I effectively leverage the distributed training system when my entire dataset doesn't fit into memory? Are there recommended strategies or examples for this scenario?

[Amyr14]

Hey. I'm not a developer of pyg, but I submitted a very similar question a while ago. You should take a look on sampling techniques. Pyg has some implemented sampling interfaces in the sampler module. If none of them are exactly what you need, you can always implement it yourself using the BaseSampler abstract class. Hope this helps.

[s-udhaya]

Hey, thanks for sharing the links. I will look into that.

[Amyr14]

Forgot to tell you that you can also use a data loader, since they implement a sampler and wrap the sampler output in a iterator that you can use to perform mini-batch training. If you still need to customize the sampling routine, you can use the NodeLoader and/or LinkLoader class and implement a BaseSampler for it.
Also, in the link you provided there is a section explaining how to use distributed data loaders.

[gabyweg76]

Hello All,

Thank you for this implementation ! I wish I could have a distributed training like that during my thesis. If only it only existed already a few years ago it could have spared me some headaches! I tried to read the documentations and I am not sure whether it exists communication between the partitions ? I saw in the partitioning there were local and remote nodes automatically created but I am not sure if for instance some communication were made between the remotes nodes for every partitions ? I wish like for CFD applications, your partitions can send their information to their neighbors, like the ghost cells/nodes so in the end we will have a smooth field without discontinuities between the partitions.

Thank you in advance

Edit: Sorry, I found it with LocalFeatureStore, it's just amazing now we can communicate information between the subgraph, I think that's the ultimate win.