Fintune whisper model returns exclamation marks

[lizgoh]

Hi, After finetuning whisper followed the blog (https://huggingface.co/blog/fine-tune-whisper#load-whisperfeatureextractor), I find eval_wer, eval_cer score are bad.
Because of that, I checked result of pred_str, about some pred_str setneces are just all exclamation marks. I checked also audio encoding part, but audio files are okay.
Is there any reason the result of finetuning returns only exclamation marks?

And these are my training arguments:
"training_args = Seq2SeqTrainingArguments(
output_dir="./whisper-small_output2", # change to a repo name of your choice
per_device_train_batch_size=16,
gradient_accumulation_steps=8, # increase by 2x for every 2x decrease in batch size
learning_rate=1e-5,
warmup_steps=50,
max_steps=4000,
gradient_checkpointing=False,
fp16=True,
tf32=True,
dataloader_num_workers=4,
evaluation_strategy="steps",
per_device_eval_batch_size=8,
predict_with_generate=True,
generation_max_length=225,
save_steps=50,
eval_steps=50,
logging_steps=1,
report_to=["tensorboard","wandb"], #, "wandb"
load_best_model_at_end=True,
metric_for_best_model="wer",
greater_is_better=False,
#push_to_hub=True,
)"

[jongwook]

It appears that the model is producing the token 0 (which gets decoded into !). I'm not familiar with the HuggingFace implementation's decoder, but it may not be configured to produce the EOT token, or the model could have forgot to do so during the fine-tuning process.

A simple workaround would be to stop sampling at the ! token, in case you don't care about the punctuations.