Whisper large multi GPU OutOfMemoryError issue #2198
-
|
Hi 👋🏼 mate... Previously i followed @sanchit-gandhi post on github as well as huggingface lastly if found two readme
deepspeed --num_gpus=4 s2s_train.py \
--deepspeed="ds_config.json" \
--model_name_or_path="openai/whisper-large-v2" \
--dataset_name="mozilla-foundation/common_voice_17_0" \
--dataset_config_name="ar" \
--language="arabic" \
--train_split_name="/home/ec2-user/SageMaker/cv-17/ar/train_data.csv" \
--max_train_samples=1000 \
--eval_split_name="/home/ec2-user/SageMaker/cv-17/ar/test_data.csv" \
--max_eval_samples=100 \
--num_train_epochs="5" \
--max_steps="-1" \
--output_dir="whisper_l2_AR" \
--per_device_train_batch_size="32" \
--gradient_accumulation_steps="2" \
--per_device_eval_batch_size="16" \
--logging_steps="25" \
--learning_rate="2.5e-5" \
--warmup_steps="3" \
--eval_strategy="steps" \
--eval_steps="100" \
--save_strategy="steps" \
--save_steps="100" \
--generation_max_length="225" \
--length_column_name="input_length" \
--max_duration_in_seconds="30" \
--text_column_name="sentence" \
--freeze_feature_encoder="False" \
--report_to="tensorboard" \
--metric_for_best_model="wer" \
--greater_is_better="False" \
--load_best_model_at_end \
--gradient_checkpointing \
--fp16 \
--overwrite_output_dir \
--do_train \
--do_eval \
--predict_with_generate
Traceback error log[INFO|trainer.py:2108] 2024-06-01 17:42:49,183 >> ***** Running training *****
Error: Note: Sometime it's showing GPU 0 something 1 or 2, base on the I'll appreciate you suggestion and idea's to solve this Good Day 👋🏼 |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 7 replies
-
|
smaller batch size |
Beta Was this translation helpful? Give feedback.
-
|
Hi @phineas-pta Thanks for your suggestion. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @sadhiin - you can likely use - --model_name_or_path="openai/whisper-large-v2" \
+ --model_name_or_path="openai/whisper-large-v3" \Assuming you've set-up deepspeed correctly (which it looks like you have based on your config), I would recommend reducing your per-device batch-size: --per_device_train_batch_size="16" \
--gradient_accumulation_steps="1" \Note that your overall batch size will be: Plugging in the numbers for your specific config: Which should be sufficient for fine-tuning. If you still hit OOM, reduce your per-device batch size by another factor of two, and increase your gradient accumulation steps by a factor of two. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you, @sanchit-gandhi . That's really helped. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, @sanchit-gandhi |
Beta Was this translation helpful? Give feedback.
-
|
@sadhiin have you finetuned it then ? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Navanit-git 👋🏼 Luckily I made that. |
Beta Was this translation helpful? Give feedback.
-
|
kindly share it on hugging face. I am searching for best arabic tts model for normal conversations and there is currently no model which can beat large v3 |
Beta Was this translation helpful? Give feedback.
-
|
I'm unable to share that, as the work is patented by my previous company 😞 |
Beta Was this translation helpful? Give feedback.
-
|
sad to hear that. Its okay thanks anyways |
Beta Was this translation helpful? Give feedback.
Hey @sadhiin - you can likely use
large-v3instead oflarge-v2, since it's a stronger pre-trained model:Assuming you've set-up deepspeed correctly (which it looks like you have based on your config), I would recommend reducing your per-device batch-size:
--per_device_train_batch_size="16" \ --gradient_accumulation_steps="1" \Note that your overall batch size will be:
Plugging in the numbers for your specific config:
Which should be sufficient for fine-tuning. If you still hit OOM, reduce…