fine tune whisper for my local language

[sheicky]

Hi, I'm working on a project where the goal is to fine-tune a model to learn my local language. I have a dataset of over 40 hours of audio and corresponding transcriptions. The challenge I'm facing is that I haven't been able to fine-tune Whisper successfully, and my current conclusion is that Whisper might not support my local language. Do you have any ideas ?

[hwanython]

what kinds of local language?

[urroxyz]

You'll need to add the language code to the tokenizer's vocabulary as <|xx|> (wherein xx is the language code of your desired language), and fine-tune from there.

Finetuning a new language is definitely possible, but a lot of data and compute is required to do it, even for the tiny model, that is, if you'd like to run a full train. I suggest PEFT or qLoRa training if you don't have that much hardware.

[hwanython]

"Would adding language tags like <|ko|> and <|en|> in code-switched sentences help the model better handle language transitions during fine-tuning or inference?"

[treeaaa]

https://arxiv.org/html/2506.00291v1