Approaches to scaling up training with millions of homogeneous graphs that don't fit into workstations memory

[br0kej]

Hey Community,

I hope you can help! I am currently researching the use of GNN's for binary function search and want to scale up my experiments. I am able to train on approximately 5 million graphs as they fit within my workstations RAM but I want to explore scaling up to use a larger proportion of my dataset (approximately 25 million graphs).

I have done some cursory research and I have seen torchdata as an option and the associated DataPipe's. From looking at the recent issues though, its basically dead in the water and I am wary of using it.

How have other folks tackled this problem?

[rusty1s]

We just introduced OnDiskDataset for such use-cases, see https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.data.OnDiskDataset.html#torch_geometric.data.OnDiskDataset.

Here is an example: https://github.com/pyg-team/pytorch_geometric/blob/master/torch_geometric/datasets/pcqm4m.py

[br0kej]

This looks great. I will have an experiment over the next week or so and report back! From reading the docs, the first stage would be to transfer my current JSON networkx format graphs into an sqlite DB right?