Partial requant feature to save compute and time during tests.

[Nexesenex]

Hey,

Could it be possible to have a partial requant feature?

For (a generic) example, one quantizes a IQ2_KT .gguf, but with ffn_down in IQ2_S and the output in IQ5_KS_R4.
Then, one wants to requantize the same model with the same IQ2_KT broad quant strategy, but with ffn_down in IQ3_XXS and the output in IQ5_K.

Could a feature be implemented so the first quantized model is used as a secondary source to the original source, in order import all the already quantized tensors in IQ2_KT from this secondary source, copy them in the destination .gguf, and only requantize from the original source those tensors which the type has been changed in the quantization command?

That could save a lot of time and compute during tests.

[saood06]

People do similar things a lot by making scripts that leveraging gguf-py. (Some notable examples was updating the gemma QAT to use use Q6_K instead of fp16 for the embeddings table, manually making deepseek R1-T chimera from a V3 and R1 GGUF, etc.).

I've thought to add support to the C/C++ code to do this, but it seems unnecessary given how flexible gguf-py is.

There has been effort made to keep gguf-py current with all the quant types (see #458 and #298).

[ikawrakow]

It would be useful, right? When I'm actively experimenting with quantization mixes I wish I had this feature. But implementing it basically means to re-implement quantization, so I have not done it.

The alternative is to run a second quantization where only the tensors that you want to change are quantized (using --custom-q), and then, as @saood06 mentions, use gguf-py to stitch the two models together (although, I don't think there is an easy out-of-the-box way of doing, or is there?)

[saood06]

I have actually thought about this before, and thought the most polished version would be to add this functionality both as a standalone script (taking in some regex similar to --custom-q, -ot, --repack-pattern, etc.) and in the GGUF Editor GUI : ggml-org/llama.cpp#12930 (which has yet to be ported here).

I never did it as it was always so easy to make one-off scripts for my gguf-py needs, and I thought it wasn't something that many other people would care about or use, but I guess I was wrong.

[Nexesenex]

Well, we are actually several folks testing new quants on different models, and so the idea might be quite popoular, ideally in C or at least in Python. I'll try by myself if no one comes with an out of the box solution, but need to read all those references and understand more about what I'm getting into, because I'm far far behind you guys about the know-how.

[saood06]

Well, we are actually several folks testing new quants on different models, and so the idea might be quite popoular, ideally in C or at least in Python.

Yeah. I floated this idea a long time ago to a certain quant maker (who pumps out a lot of quants) as it would (and still could) save them a lot of wasted compute, but this was before I knew about gguf-py.

I'll try by myself if no one comes with an out of the box solution

Nice, if you don't end up getting something working by the time I finish up polishing the frontend I use to be good enough for a public release I'll do it.

but need to read all those references and understand more

Here's two I mentioned, QAT embed swap, DIY chimera merge. I know I've seen more but these are the first two that came to mind.

[saood06]

Also I just remembered there was another hacky idea I had to do this which involved abusing the gguf-split system to isolate any tensors you want to experiment with which would allow you to swap them out (and test many combinations).

The best implementation of this could in theory minimize both the amount of space taken (should be easy) and the amount of files written to (this seems like it would be much more difficult, quantizing only select tensors with gguf-py might not be too bad, but that would limit it to only the tensors it can quantize to, and doing it with quantize.cpp means adding that functionality to it which may be difficult).

[Nexesenex]

Also I just remembered there was another hacky idea I had to do this which involved abusing the gguf-split system to isolate any tensors you want to experiment with which would allow you to swap them out (and test many combinations).

The best implementation of this could in theory minimize both the amount of space taken (should be easy) and the amount of files written to (this seems like it would be much more difficult, quantizing only select tensors with gguf-py might not be too bad, but that would limit it to only the tensors it can quantize to, and doing it with quantize.cpp means adding that functionality to it which may be difficult).

Lol, I was just thinking about this 1h ago. (Why don't I simply split the gguf in as many tensor as there is..), and then it becomes a matter of naming. I was contemplating over that a long time ago already, tensor-series based gguf, gguf as directory and so on. But actually, it can already be tried as things are.

[saood06]

@Nexesenex

Have you seen this: https://github.com/Thireus/GGUF-Tool-Suite? I haven't fully gone through the code yet, but I think it seems to accomplish at least some of the goals you described here (taking the path of using the gguf-split system).

[Nexesenex]

@Nexesenex

Have you seen this: https://github.com/Thireus/GGUF-Tool-Suite? I haven't fully gone through the code yet, but I think it seems to accomplish at least some of the goals you described here (taking the path of using the gguf-split system).

You will laugh. I discovered his fork of IKL today, and didn't discover yet his tools suite. Thanks for the heads-up, I will dive into it asap! :)

[saood06]

Thanks for the heads-up, I will dive into it asap! :)

Let me know your thoughts, e.g. if it does meet your goals, will you use it, will you change/fork it, etc.

[Nexesenex]

Thanks for the heads-up, I will dive into it asap! :)

Let me know your thoughts, e.g. if it does meet your goals, will you use it, will you change/fork it, etc.

Sure thing.