Deprecate `prepare_data_per_node` and auto detect shared file-system

[rohitgr7]

Proposed refactor

This was proposed by @tchaton on live-stream. prepare_data_per_nodecan be identified by dumping a file on rank 0 and checking whether all other ranks can see that file or not. In the case of a multi-node system, if the file system is shared then all of them will be able to see that file, else not.

Motivation

This way users won't have to take care of setting this flag manually.

Pitch

possible pseudocode (feel free to update this or comment)

if global_rank == 0:
    filename = 'something_{uuid()}.txt'
    broadcast(filename)
    dump_file(filename)

barrier()

if global_rank == 0
    is_file_present = does_file_exist(filename)
    is_file_present = all_gather(is_file_present)
    prepare_data_per_node = is_file_present.all()
    prepare_data_per_node = broadcast(prepare_data_per_node)

any other alternatives??

Additional context

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Lite: enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.

• Bolts: Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.

• Lightning Transformers: Flexible interface for high-performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

cc @Borda @justusschock @kaushikb11 @awaelchli @ananthsub @ninginthecloud @jjenniferdai @rohitgr7 @akihironitta

[justusschock]

The question is where to dump this file? Because I know that many clusters have a lot of write-restrictions to certain directories.

[rohitgr7]

default_root_dir maybe? or is there any directory such as /tmp or something that is always open in such a case, since a specific directory here is not a requirement.

[awaelchli]

What's shared is usually the home directory and some selected mount points. It's undefined where you would need to make this check. For the prepare hook for instance, when downloading data there the check would have to be performed on the folder where the data gets downloaded. It's not possible for PL to know this location, or where the user will copy the files afterwards. I would prefer not to make any assumptions and not support this "feature".