when will rtx50 series GPU be supported?

[nianqiu-liu]

thanks!

[tridao]

It's already supported

[HDANILO]

Just finished installing again, after purging:

Using /home/hdanilo/miniconda3/lib/python3.12/site-packages
Finished processing dependencies for flash-attn==2.7.4.post1
(base) hdanilo@DragonRollDev:/mnt/c/Users/helto/diffusion-pipe/flash-attention$ python -m pip list | grep flash
DEPRECATION: Loading egg at /home/hdanilo/miniconda3/lib/python3.12/site-packages/flash_attn-2.7.4.post1-py3.12-linux-x86_64.egg is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
flash_attn                               2.7.4.post1
flash_attn                               2.7.4.post1

then run the tests:

(base) hdanilo@DragonRollDev:/mnt/c/Users/helto/diffusion-pipe/flash-attention$ python -m pytest tests/test_flash_attn.py
================================================= test session starts ==================================================
platform linux -- Python 3.12.9, pytest-8.3.5, pluggy-1.5.0
rootdir: /mnt/c/Users/helto/diffusion-pipe/flash-attention/tests
configfile: pyproject.toml
plugins: anyio-4.8.0, langsmith-0.3.15, docker-3.1.2, hydra-core-1.3.2
collected 508772 items

tests/test_flash_attn.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF [  0%]
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF^C^C

one of the errors:

        return_softmax: bool
    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
        q, k, v = [maybe_contiguous(x) for x in (q, k, v)]
>       out, softmax_lse, S_dmask, rng_state = flash_attn_gpu.fwd(
            q,
            k,
            v,
            None,
            alibi_slopes,
            dropout_p,
            softmax_scale,
            causal,
            window_size_left,
            window_size_right,
            softcap,
            return_softmax,
            None,
        )
E       RuntimeError: CUDA error: no kernel image is available for execution on the device
E       CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
E       For debugging consider passing CUDA_LAUNCH_BLOCKING=1
E       Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[HDANILO]

Double checking torch functionality:

(base) hdanilo@DragonRollDev:/mnt/c/Users/helto/diffusion-pipe/flash-attention$ python
Python 3.12.9 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:27) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.cuda.is_available())
True
>>> print(torch.cuda.get_arch_list())
['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120', 'compute_120']
>>> zeros_tensor_gpu = torch.zeros((50, 50), device='cuda')
>>> zeros_tensor_gpu
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')

[nuxxor]

@HDANILO is right it's not working on RTX 5090.

[jkpjkpjkp]

is there updates on this? after following the hopper/ installation guide tests failed with CUDA error (/home/jkp/g/flash-attention/hopper/flash_fwd_launch_template.h:192): no kernel image is available for execution on the device rtx5080

[CristyNel]

I spent 3 hours building FA3, only to realize it doesn't work on the 5090 (sm120).