Global device default

[jeff-hykin]

Is there a configuration option for having all trainers default to taking full advantage of available hardware instead of having to opt-in every time for every trainer?

I searched through the Trainer docs, but wasn't able to find an option.

Although this isn't the best interface, what I roughly looking for was

import pytorch_lightning as pl
import torch

if torch.cuda.is_available():
    # option1, shared-server with 8 GPUs and multiple jobs
    pl.Trainer.defaults = dict(
        gpus=1,
        auto_select_gpus=True,
    }
    
    # option2, developer computer with only 1 job at a time
    pl.Trainer.defaults = dict(
        gpus=torch.cuda.device_count(),
        auto_select_gpus=True,
    }

On a similar note (and maybe this deserves it's own separate discussion), when inheriting from pl.LightningModule, is there some way to set self.device to default to a device other than CPU? Even if the model is sent to GPU, after a training session is run it seems the trainer forcibly sends the model back to the CPU

[staniPetrox]

On your first issue, what exactly do you mean by taking "full advantage of available hardware"? Do you mean all available GPUs?

[jeff-hykin]

I was hoping the user could define what that means depending on their project (and I think that follows the spirit of pytorch lightning being flexible while also avoiding repetitive-boilerplate code)

It could be all GPUs, just 1, a TPU, or different memory sharing options for the trainer. Regardless of which of those the user picks, always defaulting to CPU is definitely not taking full advantage of hardware.

For an example of what I had in mind;
(And I'll add this to the original post)

import pytorch_lightning as pl
import torch

if torch.cuda.is_available():
    # option1, shared-server with 8 GPUs and multiple jobs
    pl.Trainer.defaults = dict(
        gpus=1,
        auto_select_gpus=True,
    }
    
    # option2, developer computer with only 1 job at a time
    pl.Trainer.defaults = dict(
        gpus=torch.cuda.device_count(),
        auto_select_gpus=True,
    }

Another way, to avoid truly global defaults, would be to have it return a modified version of pl.

import pytorch_lightning as pure_pl
import torch

# the rest of the project would do
#    from *this_file* import pl 

if not torch.cuda.is_available():
    pl = pure_pl
else:
    pl = pure_pl.customize_defaults(
        Trainer=dict(
            gpus=1,
            auto_select_gpus=True,
        )
    )