lightning.fabric.utilities.exceptions.MisconfigurationException: No supported gpu backend found! Maybe latest gpu compability issue..?

[Shrimpwoo]

Bug description

I'm trying to run the CityGausssian project using pytorch-lightning on an RTX 5080 blackwell gpu.

However, even though you have the right GPU, Cuda Toolkit, and Pytorch version installed, I keep getting the following error: 'raise MisconfigurationException(“No supported gpu backend found!”)
lightning.fabric.utilities.exceptions.MisconfigurationException: No supported gpu backend found!' error log.

May I get any help??
thx for read this issue.

What version are you seeing the problem on?

v2.5

How to reproduce the bug

clone the CityGaussian project and just simply put this command in bash terminal.

bash ./scripts/run_citygs_mc_aerial.sh

then you got the error log.

Error messages and logs

GPU  is available.
Seed set to 42
output path: /home/bravo/3D_GS/CityGaussian/outputs/citygsv2_mc_aerial_coarse_sh2
/home/bravo/anaconda3/envs/city_gs_cu128/lib/python3.9/site-packages/lightning/pytorch/utilities/parsing.py:209: Attribute 'renderer' is an instance of `nn.Module` and is already saved during checkpointing. It is recommended to ignore them using `self.save_hyperparameters(ignore=['renderer'])`.
Traceback (most recent call last):
  File "/home/bravo/3D_GS/CityGaussian/main.py", line 4, in <module>
    cli()
  File "/home/bravo/3D_GS/CityGaussian/internal/entrypoints/gspl.py", line 12, in cli
    CLI(
  File "/home/bravo/anaconda3/envs/city_gs_cu128/lib/python3.9/site-packages/lightning/pytorch/cli.py", line 392, in __init__
    self.instantiate_classes()
  File "/home/bravo/anaconda3/envs/city_gs_cu128/lib/python3.9/site-packages/lightning/pytorch/cli.py", line 563, in instantiate_classes
    self.trainer = self.instantiate_trainer()
  File "/home/bravo/anaconda3/envs/city_gs_cu128/lib/python3.9/site-packages/lightning/pytorch/cli.py", line 577, in instantiate_trainer
    return self._instantiate_trainer(trainer_config, extra_callbacks)
  File "/home/bravo/anaconda3/envs/city_gs_cu128/lib/python3.9/site-packages/lightning/pytorch/cli.py", line 602, in _instantiate_trainer
    return self.trainer_class(**config)
  File "/home/bravo/anaconda3/envs/city_gs_cu128/lib/python3.9/site-packages/lightning/pytorch/utilities/argparse.py", line 70, in insert_env_defaults
    return fn(self, **kwargs)
  File "/home/bravo/anaconda3/envs/city_gs_cu128/lib/python3.9/site-packages/lightning/pytorch/trainer/trainer.py", line 396, in __init__
    self._accelerator_connector = _AcceleratorConnector(
  File "/home/bravo/anaconda3/envs/city_gs_cu128/lib/python3.9/site-packages/lightning/pytorch/trainer/connectors/accelerator_connector.py", line 144, in __init__
    self._accelerator_flag = self._choose_gpu_accelerator_backend()
  File "/home/bravo/anaconda3/envs/city_gs_cu128/lib/python3.9/site-packages/lightning/pytorch/trainer/connectors/accelerator_connector.py", line 354, in _choose_gpu_accelerator_backend
    raise MisconfigurationException("No supported gpu backend found!")
lightning.fabric.utilities.exceptions.MisconfigurationException: No supported gpu backend found!

Environment

Current environment

#- PyTorch Lightning Version (2.5.0.post0):
#- PyTorch Version (2.7.0.dev20250306+cu128):
#- Python version (3.9.21):
#- OS (WSL2 : Ubuntu 20.04):
#- CUDA/cuDNN version: 12.8, 9.71.26
#- GPU models and configuration: RTX 5080, 
#- How you installed Lightning(`conda`, `pip`, source): pip and source (both pip and source tried)

More info

No response

[jubsim]

I had a similar error. I fixed it by making sure I had PyTorch for CUDA 12.8 installed:

pip3 install torch --index-url https://download.pytorch.org/whl/cu128

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions - the Lightning Team!

[SkafteNicki]

This is a issue not related to lightning. The error is raised here:

pytorch-lightning/src/lightning/fabric/connector.py

Lines 331 to 333 in e55650d

	if CUDAAccelerator.is_available():
	return "cuda"
	raise RuntimeError("No supported gpu backend found!")

which calls this method

pytorch-lightning/src/lightning/fabric/accelerators/cuda.py

Lines 66 to 67 in e55650d

	def is_available() -> bool:
	return num_cuda_devices() > 0

which calls

pytorch-lightning/src/lightning/fabric/accelerators/cuda.py

Lines 143 to 145 in e55650d

	def num_cuda_devices() -> int:
	"""Returns the number of available CUDA devices."""
	return torch.cuda.device_count()

e.g torch.cuda.device_count() returns 0, which usually means that when installing torch something went wrong, most likely due to installing a version that is not compatible with the driver of the GPU.

Closing issue for now, but reopen and ping me if necessary.