pytorch lightning trainer cant find gpu

[ElinorG11]

Hello,

I'm trying to use this git repo: https://github.com/huyvnphan/PyTorch_CIFAR10 to test trained pytorch models on CIFAR. the writer of this repo used pytorch lighting package, and I'm experiencing the following error:

Traceback (most recent call last):
  File "train.py", line 83, in <module>
    main(args)
  File "train.py", line 28, in main
    trainer = Trainer(
  File "/home/elinor/aihwkit_env/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/env_vars_connector.py", line 38, in insert_env_defaults
    return fn(self, **kwargs)
  File "/home/elinor/aihwkit_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 426, in __init__
    gpu_ids, tpu_cores = self._parse_devices(gpus, auto_select_gpus, tpu_cores)
  File "/home/elinor/aihwkit_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1522, in _parse_devices
    gpus = pick_multiple_gpus(gpus)
  File "/home/elinor/aihwkit_env/lib/python3.8/site-packages/pytorch_lightning/tuner/auto_gpu_select.py", line 37, in pick_multiple_gpus
    picked.append(pick_single_gpu(exclude_gpus=picked))
  File "/home/elinor/aihwkit_env/lib/python3.8/site-packages/pytorch_lightning/tuner/auto_gpu_select.py", line 58, in pick_single_gpu
    raise RuntimeError("No GPUs available.")
RuntimeError: No GPUs available.

The line where the writer is calling Trainer (from train.py):

trainer = Trainer(
            fast_dev_run=bool(args.dev),
            logger=logger if not bool(args.dev + args.test_phase) else None,
            gpus=1,
            auto_select_gpus=True,
            deterministic=True,
            weights_summary=None,
            log_every_n_steps=1,
            max_epochs=args.max_epochs,
            checkpoint_callback=checkpoint,
            precision=args.precision,
        )

I'm running this code on linux, ubuntu 20.04 with python 3.8. the command torch.cuda.device_count() returns 1.
and those are the packages installed:

Package                 Version     
----------------------- ------------
absl-py                 1.0.0       
aiohttp                 3.8.0       
aiosignal               1.2.0       
astroid                 2.6.6       
async-timeout           4.0.0       
attrs                   21.2.0      
cachetools              4.2.4       
certifi                 2021.10.8   
charset-normalizer      2.0.7       
cmake                   3.21.4      
cycler                  0.11.0      
distro                  1.6.0       
frozenlist              1.2.0       
fsspec                  2021.11.0   
future                  0.18.2      
google-auth             2.3.3       
google-auth-oauthlib    0.4.6       
grpcio                  1.41.1      
idna                    3.3         
iniconfig               1.1.1       
isort                   5.9.3       
kiwisolver              1.3.2       
lazy-object-proxy       1.6.0       
Markdown                3.3.4       
matplotlib              3.4.3       
mccabe                  0.6.1       
multidict               5.2.0       
mypy                    0.910       
mypy-extensions         0.4.3       
numpy                   1.21.3      
oauthlib                3.1.1       
packaging               21.2        
parameterized           0.8.1       
Pillow                  8.4.0       
pip                     20.0.2      
pkg-resources           0.0.0       
pluggy                  0.13.1      
protobuf                3.19.1      
py                      1.10.0      
pyasn1                  0.4.8       
pyasn1-modules          0.2.8       
pybind11                2.8.1       
pycodestyle             2.7.0       
pyDeprecate             0.3.1       
pylint                  2.9.0       
pyparsing               2.4.7       
PyQt5                   5.15.6      
PyQt5-Qt5               5.15.2      
PyQt5-sip               12.9.0      
pytest                  6.2.4       
python-dateutil         2.8.2       
pytorch-lightning       1.5.0       
PyYAML                  6.0         
requests                2.26.0      
requests-oauthlib       1.3.0       
rsa                     4.7.2       
scikit-build            0.12.0      
scipy                   1.7.1       
setuptools              44.0.0      
six                     1.16.0      
tensorboard             2.7.0       
tensorboard-data-server 0.6.1       
tensorboard-plugin-wit  1.8.0       
toml                    0.10.2      
torch                   1.10.0+cu113
torchaudio              0.10.0+cu113
torchmetrics            0.6.0       
torchvision             0.11.1+cu113
tqdm                    4.62.3      
types-dataclasses       0.6.1       
types-requests          2.25.11     
typing-extensions       3.10.0.2    
urllib3                 1.26.7      
Werkzeug                2.0.2       
wheel                   0.37.0      
wrapt                   1.12.1      
yarl                    1.7.2     

perhaps you have any idea why am I getting this error? I tried reading issues regarding this subject, but none helped.

Thank you.

[kaushikb11]

Could you set auto_select_gpus=False and give it a try?