Convert GPTQ model to GGUF

[Nilabhra]

Hi! I was wondering if it's possible to convert a 4bit GPTQ model (converted via AutoGPTQ) to the GGUF format? Would be great if I could have some pointers to how this can be done.

[Green-Sky]

This used to (?) be a thing. But resulted in subpar models, so i strongly argue against it. Either get the same model in gguf or convert it yourself from the original files.

[Nilabhra]

Thanks for your insight!

Is there a reasoning behind it not working? From my limited understanding, GGUF is a format which is optimized for running LLMs on CPUs. GGUF formatted models can also be quantized and then stored in the GGUF format. If one has a pre-quantized LLM, it should be possible to just convert it to GGUF and get the same kind of output which the quantize binary generates.

[Green-Sky]

quantization is a lossy thing. and llama.cpp does not support gptq. the old gptq was incidentally similar enough to , i think q4_0, that adding a little padding was enough to make it work.

If one has a pre-quantized LLM, it should be possible to just convert it to GGUF and get the same kind of output which the quantize binary generates.

llama.cpp does not support gptq, so you would have to re-quantize, which is bad, since you already lost info the first quantization.

[Nilabhra]

I see, thanks for taking the time to explain this.