HOGWILD using PyTorch Lighting

[miguelalonsojr]

Are there any plans to support CPU multiprocessing on a single machine? After digging through the codebase, #1158 doesn't seem to actually speed things up on a single node the way that pytorch hogwild (https://pytorch.org/docs/stable/notes/multiprocessing.html#hogwild) does. In order to use hogwild in pytorch, the model must be shared across each process through share_memory(), which is never called in PL.

[carmocca]

Hi!

ddp_cpu is used just for debugging ddp_spawn on CPU. see: https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html#num-processes

Supporting hogwild is currently not a priority, so I can't comment much on that.