Does `Trainer(devices=1)` use all CPUs?

[MaximilienLC]

Bug description

pytorch-lightning/src/lightning/fabric/accelerators/cpu.py

Line 75 in 3740546

def _parse_cpu_cores(cpu_cores: Union[int, str, List[int]]) -> int:

cpu_cores being a list of integers will always raise an exception, which shouldn't according to the Trainer documentation/this function signature

What version are you seeing the problem on?

master

How to reproduce the bug

No response

Error messages and logs

# Error messages and logs here please

Environment

Current environment

#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):

More info

No response

cc @Borda

[awaelchli]

@MaximilienLC Do you mean this?

trainer = Trainer(
      accelerator="cpu",
      devices=[1, 3],
  )

TypeError: `devices` selected with `CPUAccelerator` should be an int > 0.

This is correct. It does not make sense to select device indices on a CPU. Device indices are meant for accelerators like CUDA GPUs or TPUs.

If you select devices=1 (int) on CPU, PyTorch will already use all cores and threads when appropriate. And devices > 1 just simulates DDP on CPU but it does not give you any benefits in terms of speed, it's only meant for debugging DDP.

[awaelchli]

If there is documentation that contradicts this, please point me to it so we can update it. Thanks!

[MaximilienLC]

pytorch-lightning/src/lightning/pytorch/trainer/trainer.py

Line 146 in 3740546

devices: The devices to use. Can be set to a positive number (int or str), a sequence of device indices

I guess this section could add that info for CPU.

So you mean devices=1 on CPU equates to using all CPUs?

[awaelchli]

So you mean devices=1 on CPU equates to using all CPUs?

Yes PyTorch will use all CPUs if it can parallelize the operation accordingly.

[jneuendorf]

I agree that this should not throw an exception assuming the current documentation is correct. For example, the device could come from an environment variable

trainer = Trainer(
    accelerator=os.environ.get("DEVICE", "cpu"),
    devices=-1,  # use all devices
)

So when deploying the code to different machines, it would break for CPU. Maybe a warning would be more adequate to notify the user that something is not optimally configured.

Would devices="auto" also use all devices?