Can I train Whisper in my language ?

[apurvns]

I tried using Whisper for my native language Gujarati but result was not good. I am new to AI so want to know can I contribute to this project by generating training data ?

[sanchit-gandhi]

Check-out this blog for fine-tuning Whisper for multilingual ASR with Hugging Face Transformers: https://huggingface.co/blog/fine-tune-whisper

It provides a step-by-step guide to fine-tuning, right from data preparation to evaluation 🤗 There'a Google Colab so you can also run it as a notebook 😉

Feel free to share your training set-up and results, happy to help!

[DavraYoung]

For any one reading this thread, the transformers timestamp predictions are out. And it seems to work very well on provided pretrained models.

In my case I need to train whisper to be able to transcribe each word with timestamps in my language:
Here is where I got so far:
Putting timestamps tokens does work during finetuning with transformers:

def convert_timestamp_to_token(timestamp, time_precision=20):
    timestamp_begin = whisper["processor"].tokenizer.all_special_ids[-1] + 1
    return timestamp_begin + int(timestamp / time_precision)

You can use this function to add timestamps between each word or sentence. But in my case the accuracy is decreasing very much, the loss never gets less than 12% (I assume its because of per word approach)
In my case:

for index, caption in enumerate(clip["captions"]):
    is_last = index == len(clip["captions"]) - 1
    start_token = convert_timestamp_to_token(caption["start"])
    end_token = convert_timestamp_to_token(caption["end"])
    text = caption["sentence"].strip()
    tokens += [
        start_token,
        *[
            token
            for token in whisper["processor"]
            .tokenizer(" " + text)
            .input_ids
            if token < end_of_text
        ],
        end_token,
    ]

In my case, after one epoch on 400hours dataset my medium model transcribes pretty well the first 7 seconds, but then starts to fail to detect the ending timestamp (I assume its because my dataset has at max 10second audios)

cc: @sanchit-gandhi

Has any one tried training whisper to detect word timestamps?

[sanchit-gandhi]

Hey @DavraYoung! Thanks for the great write-up! Really cool that the model is able to adapt to per-word timestamps during fine-tuning; this is something that I've been meaning to investigate but haven't gotten round to, so very interesting to see your approach and hear the results from your run! Weird that the loss plateaus - did it keep going down when you fine-tuned without timestamps? Wonder if just the timestamps tokens are the ones that are contributing to the loss, and the word tokens are negligible? Maybe you could have a look at the loss terms on a per-batch basis?

Regarding the audio length, that's quite interesting to hear that the model breaks after 7s. I imagine this is probably down to the distribution of your training data being < 10s. Maybe you could try concatenating audio samples to give longer inputs?

[Zhuifeng414]

Hey @giannhskp! Super cool to hear that the fine-tuned model performs well in terms of transcription accuracy, very glad to hear this! It's interesting that the timestamp predictions are degraded so heavily (essentially to random) - this suggests a 'catastrophic forgetting' effect where the model loses all knowledge about timestamp predictions upon fine-tuning with no timestamps. It might be that we have to add some timestamp prediction to the fine-tuning regime to overcome this. You could look to the OpenAI implementation to see how this is done and copy the logic across to the modelling code in Transformers.

Just to clarify your workflow, you're fine-tuning the model in transformers and then converting it back to the OpenAI format to use timestamp prediction? Or are you running everything under transformers? We should have timestamp prediction in transformers very soon! huggingface/transformers#20620

Same issue when I fine-tuned the Whisper model on a samll dataset without timestamps; the model displayed 'catastrophic forgetting'. This led to worse performance on samples beyond the fine-tuning dataset. Do you have any solution to solve it?

[sanchit-gandhi]

PEFT is more resilient to catastrophic forgetting and gives comparable performance: https://github.com/Vaibhavs10/fast-whisper-finetuning

[sutungpo]

Check-out this blog for fine-tuning Whisper for multilingual ASR with Hugging Face Transformers: https://huggingface.co/blog/fine-tune-whisper

It provides a step-by-step guide to fine-tuning, right from data preparation to evaluation 🤗 There'a Google Colab so you can also run it as a notebook 😉

Feel free to share your training set-up and results, happy to help!
@sanchit-gandhi
For fine-tuning Whisper translation, the evaluation metric should be the Word Error Rate (WER) metric, as mentioned in your article, or other metrics such as BLEU?