Errors are masked with DDP when processes are created external to lightning

[ananthsub]

🐛 Bug

Process reconciliation added recently during the error handling path has a bug:
https://github.com/PyTorchLightning/pytorch-lightning/blob/16392a7de787a1c0c0163f9[…]95f69178d5aef3a9/pytorch_lightning/plugins/training_type/ddp.py

_sync_dir is initialized only during call_children_scripts , which is not called if the processes are externally created. hence this will mask errors when training with Slurm or torch distributed elastic or other custom cluster environments

Users will see this issue instead:

torch.save(True, os.path.join(sync_dir, f"{self.global_rank}.pl")) 
TypeError: expected str, bytes or os.PathLike object, not NoneType

which is an exception that occurs during the handling of the root cause exception. this makes it trickier to debug

Expected behavior

There should be no exceptions during the exception handling path so the root error is clear to users

Environment

This is present on Lightning 1.4

[ananthsub]

One fix would be moving this line (https://github.com/PyTorchLightning/pytorch-lightning/blob/16392a7de787a1c0c0163f9995f69178d5aef3a9/pytorch_lightning/plugins/training_type/ddp.py#L182-L183)
to setup_environment() instead

@Borda @carmocca - do you know why mypy didn't catch this?

[carmocca]

@Borda @carmocca - do you know why mypy didn't catch this?

Because that part of the codebase is not analyzed

https://github.com/PyTorchLightning/pytorch-lightning/blob/1f01db8b303e647b102f48c36e91ddb17784414f/pyproject.toml#L49