Incorrect num_workers warning

[ZhaofengWu]

Bug description

This is technically not a bug but I couldn't find a more appropriate tag.

https://github.com/Lightning-AI/lightning/blob/d5ffdfac2ad4c8ff87c9895555fc21ccc482c18c/src/pytorch_lightning/trainer/connectors/data_connector.py#L222-L230
complains if the number of dataloader workers is less than the number of CPUs. This ignores when a process is only assigned to a subset of CPUs. I think the check in pytorch is more reasonable
https://github.com/pytorch/pytorch/blob/78a0ca29d939fc3017c3281730ba19ece5162f5c/torch/utils/data/dataloader.py#L532-L546

How to reproduce the bug

Launch any training process where len(os.sched_getaffinity(0)) < os.cpu_count().

Error messages and logs

pytorch_lightning/trainer/connectors/data_connector.py:236: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 80 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.

Environment

#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow): Trainer
#- PyTorch Lightning Version (e.g., 1.5.0):  1.7.7
#- PyTorch Version (e.g., 1.10): 1.12.1
#- Python version (e.g., 3.9): 3.10
#- OS (e.g., Linux): Linux

More info

No response

cc @Borda @justusschock @awaelchli

[carmocca]

It makes sense to align our logic with PyTorch's. Would you like to contribute this improvement?

[ZhaofengWu]

So pytorch's check is for an upper bound -- if the number of workers is greater than the number of (disposable) CPUs, then it's wasteful. But Lightning seems to want a lower bound here? If the value is aligned, then we would always complain unless the number is exactly the same as that value (by lightning if lower, by pytorch if too higher). That might be fine, since # workers = # disposable CPUs sounds optimal, but might be too verbose for some scenario?

A more lenient lower bound would be something that's strictly less than pytorch's upper bound, but I can't think of a way to reasonably set such a bound. Regardless, the current lower bound can be higher than pytorch's upper bound, which should be fixed.

[Saumya-ranjan]

is this issue still open?

[ZhaofengWu]

Oh I missed the thumb up from @carmocca, as I didn't get a notification. I assume that means the desired behavior is to always complain unless # workers = # disposable CPUs?

[carmocca]

Yes. In lightning we have this because it's easy to forget to set this, but there's no good heuristic for choosing an optimal default. It depends on your hardware, the selected strategy, environments like jupyter notebooks, ...

PyTorch's check is "safer" because you rarely want more workers than CPUs.

I don't know what we can do to improve our lower bound but I agree that our warning should adhere to PyTorch's upper bound.

This issue is still open and available.

[oggyfaker]

Here is my solution !!! Just using this line and paste in your code. Your problem will be solved !

from torch.multiprocessing import set_start_method
try:
     set_start_method('spawn')
except RuntimeError:
    pass