[Tests] skip nan lora tests on PyTorch 2.5.1 CPU.

[sayakpaul]

What does this PR do?

See: https://github.com/huggingface/diffusers/actions/runs/11925826067/job/33238594250?pr=9943#step:7:261. This PR skips the NaN fuse_lora() tests when detected PyTorch version is 2.5 or above and the device is CPU.

It's likely a PyTorch bug as the following passes on PyTorch 2.4.1 CPU but not on PyTorch 2.5.1 CPU. Thanks to Ben for helping to verifying.

import torch
import torch.nn.functional as F

device = torch.device("cuda:0") if torch.cuda.is_available() else torch.device("cpu")
print(f"Device: {device}, torch version: {torch.__version__}")

with torch.device(device):
    query = torch.randn(2, 4, 16, 8)
    key = torch.randn(2, 4, 16, 8)
    value = torch.randn(2, 4, 16, 8)

    hidden_states = F.scaled_dot_product_attention(
        query, key, value, attn_mask=None, dropout_p=0.0, is_causal=False
    )
    assert torch.isfinite(hidden_states).all()

    query += torch.nan
    hidden_states = F.scaled_dot_product_attention(
        query, key, value, attn_mask=None, dropout_p=0.0, is_causal=False
    )
    assert torch.isnan(hidden_states).all()

See relevant PyTorch thread: pytorch/pytorch#141128.

[sayakpaul]

@a-r-r-o-w not important for this PR but any reason this had to be rewritten for Cog?

[BenjaminBossan]

Thanks, LGTM. My only concern is that if it gets fixed in a future PyTorch version, it can easily go unnoticed and tests continue to be skipped. That's why I usually prefer pytest.mark.xfail(..., strict=True) for such cases.

[DN6]

I agree with @BenjaminBossan marking with xfail is better.

[sayakpaul]

@BenjaminBossan @DN6 how about now?

[BenjaminBossan]

Thanks, LGTM.

[sayakpaul]

@DN6 okay to merge, pinging because of the CI failures across the board.