deepspeed erro

[LKELN]

When I try to fine-tune Llama 7B using DeepSpeed stages 1, 2, and 3 with the llm2clip run.sh script, I don't observe a significant decrease in the video memory usage. Could you please fix this issue?

[raytrun]

I've noticed this too, but I'm not sure if it's a bug. It might be because the number of parameters in the ViT model involved in training is not large, resulting in similar GPU memory usage of different DeepSpeed stages. LKELN ***@***.***> 于2025年3月19日周三 10:48写道：

…

When I try to fine-tune Llama 7B using DeepSpeed stages 1, 2, and 3 with the llm2clip run.sh script, I don't observe a significant decrease in the video memory usage. Could you please fix this issue? — Reply to this email directly, view it on GitHub <#40>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJPK6G2KUKERWRC3NRSNAPL2VDLIZAVCNFSM6AAAAABZJVWK3OVHI2DSMVQWIX3LMV43ASLTON2WKOZSHEZTAMZQGI2DSMA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> [image: LKELN]*LKELN* created an issue (microsoft/LLM2CLIP#40) <#40> When I try to fine-tune Llama 7B using DeepSpeed stages 1, 2, and 3 with the llm2clip run.sh script, I don't observe a significant decrease in the video memory usage. Could you please fix this issue? — Reply to this email directly, view it on GitHub <#40>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJPK6G2KUKERWRC3NRSNAPL2VDLIZAVCNFSM6AAAAABZJVWK3OVHI2DSMVQWIX3LMV43ASLTON2WKOZSHEZTAMZQGI2DSMA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[LKELN]

No, I try to finetune the Llama 7B model, and this will also lead to no decrease in the video memory usage.