How to finetune DeepSeek-v2 236B

[Jiayi-Pan]

Thank you for the great repo.
Do you have any suggestions on how to finetune DeepSeek-Coder v2 236B.
Is it possible to do fsdp + qlora for this moe model? What's them minimum required to do finetuning?

[NanoCode012]

We have a qlora fsdp config. You can give it a try here but change the model https://github.com/axolotl-ai-cloud/axolotl/blob/main/examples/deepseek-v2/qlora-fsdp-2_5.yaml . I'm not sure if we tested that moe model yet and require any specific change.

[NanoCode012]

We have added example configs for deepseek_v2 models. Since it works for the smaller Lite model, it should work for the larger too. https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/deepseek-v2

[ehartford]

I would also love to see an example that works.

I've been unsuccessful at this.