can dataparallel or distributed dataparallel bring performance speed up for training graph classification tasks?

[wzsunshine]

Hi all,

I have one general question is that, when training the GNNs on multi-GPU, will it have performance speedup, on what condition it will have or not? Any hint will be appreciated.

[rusty1s]

This depends and is best evaluated empirically. If you can already process a large batch size on a single GPU (e.g., 256 or 512), multi GPU might not be worth it. However, in case of smaller batch sizes, multi GPU training might resolve in better stochastic gradients.

[rusty1s]

Thanks for the notice. This is indeed wrong and I fixed it in the example. The dataset will now get solely downloaded and processed by the main process :)

[wzsunshine]

It seems still have problems here.
Let's use 4 GPUs!
Traceback (most recent call last):
File "distributed_batching.py", line 147, in
File "python3.7/site-packages/torch/multiprocessing/spawn.py", line 200, in spawn
File "python3.7/site-packages/torch/multiprocessing/spawn.py", line 149, in start_processes
File "python3.7/multiprocessing/process.py", line 112, in start
File "python3.7/multiprocessing/context.py", line 284, in _Popen
File "python3.7/multiprocessing/popen_spawn_posix.py", line 32, in init
File "python3.7/multiprocessing/popen_fork.py", line 20, in init
File "python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
File "python3.7/multiprocessing/reduction.py", line 60, in dump
File "python3.7/site-packages/torch/multiprocessing/reductions.py", line 321, in reduce_storage
RuntimeError: unable to open shared memory object </torch_76557_2314048240> in read-write mode

[rusty1s]

Ok, thanks for letting me know. The error should now be fixed :)

[wzsunshine]

Thanks for quickly fixing！Because there is less document to discuss the multi-GPU support on PyG and I am a newcomer to this field, I have the following questions here:

1. Could you please explain the difference between distributed_sampling and distributed_batching approaches?
2. Does that have limitations on different tasks, like node classification and graph classification. With different tasks, how the batch of data (one batch size of data) will be made?
3. Is there have conditions that one task can not be trained on single GPU? For my current experience, I found that the examples in PyG did not cause out of memory issues and the training time is not a problem, which seems that data parallel will not have much benefits.

Thanks!

[rusty1s]

1. Yes, this is not particular clear. distributed_batching uses multi-GPUs to train on a set of small graphs in mini-batch mode (graph classification), while distributed_sampling uses multi GPUs to train on a single giant graph (node classification) and therefore needs to make use of sampling techniques for creating mini-batches.
2. Yes, exactly.
3. Most tasks can be trained on a single GPU, and multi GPU training might be only important in case you want to train on a really big dataset (e.g., > 1 million graphs or >10 million nodes in a single graph).