Reproducibility Issue

[VarunGumma]

I see some discrepancy in the model training script and the paper training details. It would really helpful if someone from the team can clarify these:

• The script uses "qkv_proj,o_proj,gate_up_proj,down_proj,k_proj,q_proj,out_proj,v_proj" as the target lora modules, but Qwen2 does not have qkv_proj, gate_up_proj but instead qkv proj gate_proj and up_proj modules. Is this a typo? What exact modules were trained with LoRA as I wish to reproduce the result and I am running into some issues with it.
• The script mentions that the lora scaling $\alpha$ is 64 (default), but the paper mentions it as 32.

[memray]

Hi @VarunGumma , good point, we didn't check the lora module names very carefully. Did you observe any performance diff with the correct lora modules?
In terms of alpha, I believe 64 is the correct number. We'll correct the number in the paper. Thanks for bringing it up.