How can I finetune whisper ASR and use that model in Inference?

[cod3r0k]

Hi, I have a problem in finetuning whisper ASR version large-v3 which I cant use it in inference. Do you have code for fine-tuning and using fine-tuned model in inference?
As an example, you would like to fine-tune it with a common voice

[phineas-pta]

https://huggingface.co/blog/fine-tune-whisper

[remic33]

@phineas-pta I tried it but at evaluation, the compute_metrics function seems broken and can't call tokenizer.
Not sure if the tutorial is still on point

[phineas-pta]

cannot help with very few info

[remic33]

@phineas-pta It took me time but I did the training tutorial of Huggingface for training whisper and it did not work has i was saying.
First, you have to precise the model language because batch validation on multiple language cannot be done. You can had that in model options.
When this is changed, the tokenizer called by compute metrics called do not work.
Here is my error :

`---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[31], line 1
----> 1 train_result = trainer.train()

File /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:1624, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
1622 hf_hub_utils.enable_progress_bars()
1623 else:
-> 1624 return inner_training_loop(
1625 args=args,
1626 resume_from_checkpoint=resume_from_checkpoint,
1627 trial=trial,
1628 ignore_keys_for_eval=ignore_keys_for_eval,
1629 )

File /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:2029, in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
2026 self.state.epoch = epoch + (step + 1 + steps_skipped) / steps_in_epoch
2027 self.control = self.callback_handler.on_step_end(args, self.state, self.control)
-> 2029 self._maybe_log_save_evaluate(tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval)
2030 else:
2031 self.control = self.callback_handler.on_substep_end(args, self.state, self.control)

File /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:2412, in Trainer._maybe_log_save_evaluate(self, tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval)
2410 metrics = None
2411 if self.control.should_evaluate:
-> 2412 metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
2413 self._report_to_hp_search(trial, self.state.global_step, metrics)
2415 # Run delayed LR scheduler now that metrics are populated

File /usr/local/lib/python3.10/dist-packages/transformers/trainer_seq2seq.py:166, in Seq2SeqTrainer.evaluate(self, eval_dataset, ignore_keys, metric_key_prefix, **gen_kwargs)
164 self.gather_function = self.accelerator.gather
165 self._gen_kwargs = gen_kwargs
--> 166 return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)

File /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:3229, in Trainer.evaluate(self, eval_dataset, ignore_keys, metric_key_prefix)
3226 start_time = time.time()
3228 eval_loop = self.prediction_loop if self.args.use_legacy_prediction_loop else self.evaluation_loop
-> 3229 output = eval_loop(
3230 eval_dataloader,
3231 description="Evaluation",
3232 # No point gathering the predictions if there are no metrics, otherwise we defer to
3233 # self.args.prediction_loss_only
3234 prediction_loss_only=True if self.compute_metrics is None else None,
3235 ignore_keys=ignore_keys,
3236 metric_key_prefix=metric_key_prefix,
3237 )
3239 total_batch_size = self.args.eval_batch_size * self.args.world_size
3240 if f"{metric_key_prefix}_jit_compilation_time" in output.metrics:

File /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:3520, in Trainer.evaluation_loop(self, dataloader, description, prediction_loss_only, ignore_keys, metric_key_prefix)
3516 metrics = self.compute_metrics(
3517 EvalPrediction(predictions=all_preds, label_ids=all_labels, inputs=all_inputs)
3518 )
3519 else:
-> 3520 metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
3521 else:
3522 metrics = {}

Cell In[28], line 10, in compute_metrics(pred)
7 label_ids = pred.label_ids
9 # replace -100 with the pad_token_id
---> 10 label_ids[label_ids == -100] = tokenizer.pad_token_id
12 # we do not want to group tokens when computing the metrics
13 pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)

NameError: name 'tokenizer' is not defined`
Following that code : https://github.com/bofenghuang/community-events/blob/bh/whisper-fine-tuning-event/finetune_whisper.ipynb

I am trying to finetune whisper-large-v3

[phineas-pta]

so u didnt declare tokenizer in ur code ?