Fine tuning whisper-large-v3 ASR on Punjabi/Panjabi language

[jsaluja]

Context

I finetuned whisper-large-v3 following instructions on https://colab.research.google.com/github/sanchit-gandhi/notebooks/blob/main/fine_tune_whisper.ipynb

My training dataset is 8000 examples, 30 hours of native Punjabi speaker reading Punjabi text.

I generated inferences for the training dataset on fine tuned and base model.

Problem

The inferences are exactly the same for fine tuned and base model

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 50
• num_epochs: 1
• mixed_precision_training: Native AMP

Training results

Training Loss | Epoch | Step | Validation Loss

---

0.1744 | 0.26 | 250 | 0.0974

0.0788 | 0.52 | 500 | 0.0747

0.0637 | 0.78 | 750 | 0.0582

Questions

• Does fine tuning whisper v3 work well Punjabi ?
• Is 8000 examples, 30 hours sufficient for fine tuning ?

[jsaluja]

Problems with Panjabi ASR on whisper-large-v2

I fine tuned whisper-large-v2 on the same Punjabi dataset.
Next, I generated inferences by invoking pipeline on both finetuned model and base model.

I invoked piplines with language in kwargs as

1. 'Panjabi'
2. '<|pa|>'

Issues

1. Both fine tuned, and base model returned precisely the same predictions.
2. 99% of inferences are generated in Urdu

It appears that

1. Model did not learn anything from fine tuning
2. Whisper v2 ignores the language kwargs even when explicitly set
3. Whisper v2 wrongly detects language Urdu for Panjabi audio

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 50
• num_epochs: 1
• mixed_precision_training: Native AMP

Training results

Training Loss | Epoch | Step | Validation Loss

---

0.2262 | 0.21 | 200 | 0.1239

0.0988 | 0.41 | 400 | 0.0758

0.072 | 0.62 | 600 | 0.0636

0.0647 | 0.83 | 800 | 0.0548

Framework versions

• PEFT 0.8.2
• Transformers 4.37.2
• Pytorch 2.1.2
• Datasets 2.15.0
• Tokenizers 0.15.1

Questions

• Does finetuning whisper work well on Panjabi language at all ? Any success in real world ?
• Is the detection of incorrect language - Urdu instead of Panjabi a known bug in V2 ?
• What Punjabi data sources did whisper-v2 train on ?
• What is the right way to finetune Whisper V2 to generate Panjabi transcriptions

[itaipee]

@jsaluja how did you resolved all these issues ?

[jsaluja]

@itaipee This worked for Panjabi.
Replace the LANGUAGE , LANGUAGE_CODE for your language

LANGUAGE = "Panjabi"
LANGUAGE_CODE = "pa" 
TASK = 'transcribe'

tokenizer = WhisperTokenizer.from_pretrained(BASE_MODEL, language=LANGUAGE, task=TASK)
processor = WhisperProcessor.from_pretrained(BASE_MODEL, language=LANGUAGE, task=TASK)
model = WhisperForConditionalGeneration.from_pretrained(BASE_MODEL, load_in_8bit=True, device_map="auto")

model.config.forced_decoder_ids = processor.tokenizer.get_decoder_prompt_ids(
    language=LANGUAGE, task=TASK
)
model.config.suppress_tokens = []

model.generation_config.forced_decoder_ids = processor.tokenizer.get_decoder_prompt_ids(
    language=LANGUAGE, task=TASK
)
model.generation_config.language = LANGUAGE_CODE
model.generation_config.suppress_tokens = []

[harveenchadha]

@jsaluja are you open to sharing this dataset ?