How to make a GNN model scalable?

[hkim716]

Suppose that I have ~20,000 big sized graphs, each graph has ~1,500,000 nodes.
I made a autoencoding model for node feature predictions. To decrease the training time, I used mini-batching with many num_workers., but the training time was about ~100 hours, even with Tesla GPU computing, which makes it difficult to fine tunning and optimize parameters.

In this case, what would you try more on this to dramatically decrease the computing time?
I saw graph sampler technique, but I'm not very familiar with it. Does it look helping a lot?
In which case we might use graph sampler, and what are the pros and cons?

[rusty1s]

This is a very challenging problem. In general, you should always use graph sampling to speed up conversion, but currently all our samplers (like NeighborLoader) assume a single graph. One alternative way is to pre-process your graphs into smaller subgraphs beforehand, and then try to autoencode this subgraphs instead.

[rusty1s]

It accelerates the forward and backward passes since you are now doing stochastic gradient descent rather than full-batch gradient descent. Depending on the graph size, a GNN on the full graph is expensive to run, and thus mini-batching can save both runtime and memory.

[hkim716]

Thanks Matt. You said current samplers assume a single graph. I looked at the Reddit example, where the dataset has a single large sized graph. Based on your explanation, I think current samplers are applicable to that type of dataset.

I have a question here. Suppose that we have an architecture using MNIST superpixel dataset with an architecture in mnist_graclus.py. Is it possible to apply neighbor sampling to speed up the training?
In that case, the training, validation, and test sets are on different graphs. The dataset consists of multiple graphs that are independent from each other. I wonder if we could use samplers with the pooling layers.

[rusty1s]

I think it is important to differentiate here between (1) mini-batching in single large graphs (where you sample subgraphs to learn from), and (2) mini-batching of small graphs (in which case you combine different graphs from a dataset to form a mini-batch). The former is supported via different sampling techniques, e.g., NeighborLoader, while the second one is supported through DataLoader. For MNIST, there is no need to apply any neighbor sampling strategy as graphs are small.

[hkim716]

I saw reddit dataset has a single very large graph, and the model used NeighborLoader with masks in the dataset, which maybe correspond to case (1). Does that means that the first case (1) is used for transductive learning, and the second case (2) is used for inductive learning?
Then in the case of predictive tasks that use similar dataset like MNIST superpixel, regardless of the size of each graph, mini-batching should be used through DataLoader, right?

[rusty1s]

That is mostly correct. For Reddit, you would always want to use NeighborLoader since it is just a single graph. The difference between transductive and inductive decides which nodes you allow to be seen during training. For the inductive setting, you would do

train_data = data.subgraph(data.train_mask)

loader = NeighborLoader(train_data, ...)