forced_decoder_ids will be ignored in favor of language=en.

[RitchieP]

Hi, I was previously following a tutorial about Fine Tuning Whisper Model but ran into a few errors when trying to train the model. The first error encounted led me to this issue on GitHub.

After I followed the answers to the issue, I added the line py model.generation_config.language = "en". I'm now getting a message

You have passed language=en, but also have set forced_decoder_ids to [[1, None], [2, 50359]] which creates a conflict. forced_decoder_ids will be ignored in favor of language=en.

Whenever my training goes through the first saving and evaluation step. I'll paste some relevant code below:

Code

from transformers import WhisperForConditionalGeneration

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")

model.generation_config.language = "en"

from transformers import Seq2SeqTrainingArguments

training_args = Seq2SeqTrainingArguments(
    output_dir="./whisper-small-eng-gen",  # change to a repo name of your choice
    per_device_train_batch_size=16,
    gradient_accumulation_steps=1,  # increase by 2x for every 2x decrease in batch size
    learning_rate=1e-5,
    warmup_steps=500,
    max_steps=2000,
    gradient_checkpointing=True,
    fp16=True,
    evaluation_strategy="steps",
    per_device_eval_batch_size=8,
    predict_with_generate=True,
    generation_max_length=225,
    save_steps=550,
    eval_steps=550,
    logging_steps=25,
    report_to=["tensorboard"],
    load_best_model_at_end=True,
    metric_for_best_model="wer",
    greater_is_better=False,
    push_to_hub=True,
    ignore_data_skip=True,
    do_eval=True
)

from transformers import Seq2SeqTrainer

trainer = Seq2SeqTrainer(
    args=training_args,
    model=model,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    tokenizer=processor.feature_extractor,
)

import transformers

transformers.logging.set_verbosity_info()

trainer.train()

Output

Context

I'm using Kaggle notebooks to train the model, the model is a Whisper-Small model and datasets are pulled from Huggingface. I also tried to go through the documentation on Huggingface and did not find it helpful to my issue.

Any help or pointers would be very much appreciated. Thanks!

---

Update

After following the discussions on Huggingface's transformer Pull Request #28687, I added these lines

model.generation_config.language = "<|en|>"
model.generation_config.task = "transcribe"

A similar error still shows up saying

You have passed task=transcribe, but also have set forced_decoder_ids to [[1, None], [2, 50359]] which creates a conflict. forced_decoder_ids will be ignored in favor of task=transcribe.

[phineas-pta]

that issue u mentioned is already resolved, update transformers and follow official fine-tuning guide

[RitchieP]

Hi, I'm pretty sure my transformers is up-to-date at version 4.38.2. And the message still shows up. The message seems to be always repeating itself stuck in an infinite loop.

May I also know which official fine-tuning guide you are referring to?

[Mattral]

I hope this one would help

huggingface/transformers#21937

in this medium article:

https://medium.com/@bofenghuang7/what-i-learned-from-whisper-fine-tuning-event-2a68dab1862

'''

You will find some defined arguments in Whisper model such as forced_decoder_ids and suppress_tokens. These arguments are defined in GenerationConfig for the generation task. However, we override these arguments during the training in order to let the model learn them by itself.

We also disable the use_cache feature in the Whisper decoder. It allows us to re-use the computed key and values of the self-attention and the cross-attention blocks to speed up the current decoding step. However it’s incompatible with the gradient checkpointing which will be applied in a later step to reduce the memory footprint.
'''

or if you wanna trace, I see line 1105 and 1110 from here:

https://fossies.org/linux/transformers/src/transformers/models/whisper/generation_whisper.py

[RitchieP]

Hi @Mattral, thanks for all the links! I also went into the generation_whisper.py file to look at what was happening. At the moment it looks like this is expected behavior, at least I guess.

Looks like when it hits the evaluation step, it will loop through the evaluation dataset until it evaluates everything inside. Hence, every time it loops, the message is displayed. So the message

You have passed task=transcribe, but also have set forced_decoder_ids to [[1, None], [2, 50359]] which creates a conflict. forced_decoder_ids will be ignored in favor of task=transcribe.

should just be ignored.

Feel free to correct me if anything is wrong.

[Mayuresh-MLE]

Hi @RitchieP I am getting same error during the inference and it breaks the loop. I am not able to get results from a finetuned model.. My primary task is to translate indic speech to english text.. so what change do i have to make as my data is multilingual and output has to be in english.
will adding the following lines in training loop solve this?
model.generation_config.language = "<|en|>"
model.generation_config.task = "translate"
Here, the language param is the input language of audio file right? (correct me if i am wrong), so in my case what should i put here?
i had previously experimented by adding following to the code but no success : !

    model.config.forced_decoder_ids = None
    model.config.suppress_tokens = []

[RitchieP]

@Mayuresh-MLE It's been quite a while since I worked on the same project. I don't think I can answer your question correctly. But from what I recall when I was working on the project, the message is not actually an error, so I just kept the model training going and eventually it did finish.