Why torch geometric only supports DDPSpawnStrategy (not the DDP) for pytorch lightning?

[saedrna]

Hi,

I have written my own PyTorch lightning module and fed the torch_geometric.loader.Dataloader to the module. Passed it to the pl.Trainer using strategy='ddp'. It seems that this pipeline trains without error.

So why in the documentation, we have the following note?

Currently only the pytorch_lightning.strategies.SingleDeviceStrategy and pytorch_lightning.strategies.DDPSpawnStrategy training strategies of [PyTorch Lightning](https://pytorch-lightning.readthedocs.io/en/latest/guides/speed.html) are supported in order to correctly share data across all devices/processes:

What data is shared? I see that the only difference from torch_geometric.loader.Dataloader to classical Pytorch Dataloader is the Collater. However, I debugged it a little, because my input is a Data list. It seems good enough for multiprocessing running.

Thanks,
Han

[rusty1s]

The problem with standard strategy='ddp' is that the dataset will get loaded into memory and duplicated for every GPU process. The ddp_spawn strategy avoids this: It will take care of moving the dataset into shared memory.

I described this here in more detail.

[saedrna]

Thanks for your reply. I am aware of this drawback of ddp. Sometimes, the additional memory consumption may be affordable. But the pickleable requirements for ddp_spawn has no workaround.
In addition, even considering the drawback, we probably should not give the death penalty to ddp. A warning in red alerts rather than throwing an error may be good enough.
In addition, ddp is sometimes a more preferable approach ("It is STRONGLY recommended to use DDP for speed and performance." quote pytorch lightning), especially for the non-InMemory datasets.

[rusty1s]

I don't think this is necessarily true. Although PyTorch Lightning recommends ddp over ddp_spawn, ddp_spawn is the recommended approach from PyTorch. Importantly, ddp improves speed but lacks flexibility.

I am okay with just outputting a warning when using ddp mode rather than throwing an error, but at least for learning on single giant graphs via LightningNodeData we need to make sure that people understand the implications. If you would like to change this, I am happy to accept a PR.