About Vulcan on AMD

[Danik-droid]

ggml-org#10710

There is a bug in Vulkan on AMD cards and it will probably stay until AMD fixes the drivers - maybe someday, maybe never.

I propose to put notes that Q8 does not work properly with AMD with Vulkan. Other Quants with L (Q8_0 for embed and output weights) work without any problems.

I haven't noticed any problems with other quants, new fixes improved performance by 2-3 t/s, nice!

Radeon 6900xt
koboldcpp-1.87
Windows 10 with new AMD drivers

[henk717]

Are you sure this is a thing on our latest releases? Last release we had used an older version of the vulkan code to prevent this from happening. Right now we should have ggml-org#12472

[LostRuins]

I think occam found a dp4a bug, not sure if its related but i'll do a patch release once its fixed and hopefully that will solve it

[Danik-droid]

Yes I am sure Q8, I checked Llama 3, LLama 3.1, Mistral Nemo, Mistral 7B, Gemma 2/3. It generates nonsense responses on my radeon 6900xt.

So far I haven't noticed any problems with other quants.

Edit: And yet, in other quants sometimes it causes AMD driver error. Now with the context above 4k the controller crashes all the time in Q6, Q5, Q4. Previous version of koboltcpp works fine.

[LostRuins]

Hi @Danik-droid, Sorry for the trouble. Can you try 1.87.2 and see if it's fixed?

[Danik-droid]

1.87.2, first tests: Q8 from different models one after another, all working. Above the 4k context it works correctly, I think everything is fine. No GPU driver stability issues.

I'll test it longer this weekend, maybe even tomorrow. If something is wrong I'll create a new topic, you can close this one - the problem seems to be solved.

Thank you very much.