how to training each input graph is too big to a single GPU?

[choiwanuk]

Hi, I'm going to training a custom GCN layers by a training set with pytorch.
The problem is that, each input graph is too big to input in a one graphic card.[My graphic cards 3090 * 2]
I'm searching methods(dataparallel, distributedparallel, DGL ..etc) for this situation.
But I think that this case is not solved with those methods..
By which way, can I treat this problem?
(Neugraph is the only one Method I can use?..)

[rusty1s]

How big is your graph? We provide several scalability techniques to enable learning on large graphs, e.g.:

• NeighborSampler
• Cluster-GCN
• GraphSAINT

[choiwanuk]

It is a CFD particle graph.( ~300K nodes, 40 edges per nodes and some features.)
I learning with smaller CFD cases and assumed that original case may take more than 30GB Graphic Memories.
With using those techniques, doesn't need to put entire graph on GPU? -> I heard little bit about Cluster-GCN [partitioning] but others are first time that I heard.

[choiwanuk]

I thought that this problem only be solved with GPU techniques[MPI ..]

[rusty1s]

You can well train this graph on a single GPU (although multiple GPUs don't hurt ofc), but you still need same GNN scalability techniques going from full-batch training to mini-batch training. The most famous approach for doing this is via NeighborSampler (see examples/reddit.py for an example), where we sample k-hop neighboring nodes of a mini-batch to learn from.