Question about v1.29's no CLBLAST for K-quants

[7erminalVelociraptor]

So I understand that the new K-quantization is only supported by openblas for now. Without CLblast does that also mean no GPU layers offloading?

[LostRuins]

That is correct. Until the dequantize and mat_mul kernels are implemented, K-quants will not work with CLBlast.

You can also follow this issue ggml-org#1725

Someone will have to come up with the implementation as I am not familiar enough with CL kernels to do them correctly. I'm not sure why the person who added k-quants didn't add a CL implementation as well.

[7erminalVelociraptor]

I think I could have phrased my question better. I was wondering if we can (for now) still keep GPU layer offloading without CLBlast

[LostRuins]

Unfortunately no, CLBlast is required for GPU offloading.

[LostRuins]

If anyone is able to write CL kernels, contributions (either here or preferably upstream) are greatly welcome

[cgessai]

I can't tell, is this announcement from today a solution to the issue we're talking about?

[Thellton]

I can't tell, is this announcement from today a solution to the issue we're talking about?

Unfortunately not. CUDA is a proprietary API that is only available on Nvidia GPUs. this will likely end up coming down stream in time to Koboldcpp due to it being a fork of LLaMAcpp so for anybody with a Nvidia GPU, they've got that to look forward to. for those of us with AMD and Intel GPUs... we're just going to have to watch the q2_K and q3_K issues over at LLaMAcpp for movements :(

[cgessai]

Ah, thank you for the additional clarification!

[LostRuins]

Implemented in newest version.