Choose rank check based on filesystem

[carmocca]

This could be an issue in the future if the node's filesystem becomes one a single volume.

Ideally, I think we should detect if the filesystem is shared. The condition would become (if self.global_rank if fs.shared_filesystem else self.local_rank)

Originally posted by @tchaton in #11070 (comment)

Refers to this check:

https://github.com/PyTorchLightning/pytorch-lightning/blob/7e10f6d41fee5c1ac4dceab8abc0177955239094/pytorch_lightning/plugins/training_type/tpu_spawn.py#L335-L337

cc @Borda @awaelchli @rohitgr7 @akihironitta @tchaton @kaushikb11

[awaelchli]

The detection could be done by writing a file on rank 0 and asking all others to read it with a small timeout. Not aware of any other way.

[tchaton]

@awaelchli Yes, I believe this is the only way. However, which component should be responsible for this mechanism?

I don't believe it can be CheckPointIO as other components might need to know this information.

Any ideas?

[awaelchli]

The mechanism could be implemented purely in a function and be called by the strategy. I think it should be the strategy because it is generally responsibly for the distributed communication. The check would have be performed only once, perhaps during setup()?

[tchaton]

Yes, I think this sounds good to me. This would be used by DDP and DDPSpawn.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!