accuracy_recovery_adapters

[Manni1000]

I think it would be very cool if the NF4 version had an accuracy recovery adapter. It might increase the quality to the level of the INT8 model without increasing the RAM/VRAM usage by much.

Here is some info about the idea:

https://huggingface.co/ostris/accuracy_recovery_adapters
https://www.reddit.com/r/LocalLLaMA/comments/1mytbfz/accuracy_recovery_adapter_with_selfgenerated_data/

I think it's originally from Apple for on-device models. It's similar to quantization-aware training.

I wonder how expensive it would be to make one for this model. What do you think?

[EricRollei]

It's an interesting idea. Another poster suggested that the Instruct NF4 quality wasn't very good. I plan to do some Hunyuan Image 3 training so maybe can also look at the recovery adapter. I

[EricRollei]

I have discovered that I missed one section that should stay bf16 in my quant scripts. Redoing those now, should improve quality some. Is there's still a big gap then I'll look at the recovery adapter. I may be able to train on my machine with rtx6000 and rtx 5000