Flash Attention on 1.83.1 breaks Mistral Small 3 models

[Magiwarriorx]

Enabling Flash Attention on 1.83.1 with a Mistral Small 3 model leads to the model replying with Unicode garbage when the prompt exceeds 4k tokens, regardless of context size (prompts only slightly over 4k tokens can produce passable results occasionally, try like 8k tokens for a clear picture). Disabling Flash Attention, or using 1.82.4, fixes the issue.

Additional Information:
Windows 10, AMD 7800X3D, RTX 4090, latest Nvidia drivers

[LostRuins]

What backend are you using? CUDA or Vulkan?

[Magiwarriorx]

CUDA. Hadn't thought to try Vulkan until you asked, but seemingly does not affect Vulkan. Affects both the cu12 and non-cu12 releases.

[LostRuins]

I cannot repro it on my end, but I have heard other users mentioning something similar too. It's almost definitely an issue from the upstream cuda code changes, but I have not managed to isolate which commit causes it. On my own RTX4090 laptop, it works fine.

Perhaps if you can reproduce this problem in llama.cpp you could open an issue there.

[Noircogi]

It seems there's some sort of general corruption happening under Cuda in 1.83.1 and 1.84. At a certain point, the model seems to be responding to old inputs. Reverting to 1.82.4 with the same configuration and session data works fine.

[LostRuins]

@Noircogi could you kindly share your launch setup as well as system specs? I may have to rent a GPU to try and reproduce this issue.

[LostRuins]

Preferably share a kcpps config and json save file to repro the issue

[Noircogi]

This system is a 12900K with a 4090 and a 3080TI, 64GB ram, running Windows 11
CydoniaV2.zip
Model is https://huggingface.co/TheDrummer/Cydonia-24B-v2-GGUF Q6_K
Used with Mistral V7 settings under ST.
It will go bad fast with most character cards.

[LostRuins]

I can confirm this is an issue, but somehow only on the 4090 Desktop version. I will look into it again when I can, please stand by. For now, 1.82.4 should work.

[LostRuins]

I had a fix prepared, but I think ggml-org#11894 fixes it better.

[LostRuins]

Please try the latest build 1.84.2

[Magiwarriorx]

Seems to fix the issue on my end. Thank you!