Full finetuning for 8B model on 24GB card?

[fzyzcjy]

Hi thanks for the tool! I wonder whether it is possible to full finetune a 8B model on 24GB GPU using this tool? (Can have cpu offloading, surely)

[NanoCode012]

Hey, I don't think it's possible with 24GB. Would you be able to try with 2x48GB with this config https://github.com/axolotl-ai-cloud/axolotl/blob/main/examples/llama-3/fft-8b-liger-fsdp.yaml?

[fzyzcjy]

Yes, lora/qlora looks pretty well. However, it seems that their outcome is not as good as full finetune.

[NanoCode012]

If you want to experiment, you can try spectrum https://github.com/axolotl-ai-cloud/axolotl/tree/main/src/axolotl/integrations/spectrum where you only unfreeze certain layers at a time and train them before repeating it for other layers.

You can see an example of this for llama in this issue #1991 . The only part you need to see is unfrozen_parameters:

Personally, I haven't tried it for such purpose before and very unsure if it will still fit within 24gb.

But I do not have 2x48GB cards :(

If cloud compute is an option, we do have docker image support and instructions for them. It is convenient if you're only running for a short while and need beefy compute.

[fzyzcjy]

Thank you for the suggestions!

Still I hope for full finetuning, while freezing some layers looks like will result in suboptimal behaviors...

Anyway, thanks again for your help :)

[NanoCode012]

In this case, how about considering a smaller model? If you're using it for a specialized task, fine-tuning them may be another approach.

[fzyzcjy]

Yes, currently I am playing with ~1B models. But since most papers use 7B, I would love to also have a try on that.