[ENHANCEMENT] Do you have a plan that supports Mixtral 8x7B?

[matrixssy]

[cdj0311]

The same question.

[matrixssy]

The same question.

I am working for it, but I am not sure if it will be accepted.

[bityigoss]

@matrixssy any progress ? thank you.

[matrixssy]

see #667

[yanring]

Hi, Please refer to this script for MoE/Mixtral training.

[matrixssy]

Hi, Please refer to this script for MoE/Mixtral training.

Great! But I would like to know how to convert Hugging Face (HF) weights to Megatron (MG) format, and if it's possible to convert them back after training?

[github-actions]

Marking as stale. No activity in 60 days.