I'm getting Arabic script transcription instead of Somali script for a Somali audio

[maugas]

import whisper
model = whisper.load_model("medium")
audio = whisper.load_audio("0001.wav")
audio = whisper.pad_or_trim(audio)
mel = whisper.log_mel_spectrogram(audio).to(model.device)

detect the spoken language

_, probs = model.detect_language(mel)
print(f"Detected language: {max(probs, key=probs.get)}")
Detected language: so

#options = whisper.DecodingOptions()
options = whisper.DecodingOptions(language = 'so', task='transcribe')
result = whisper.decode(model, mel, options)

print(result.text)

• ساكا سيدانكو سوبي ملاين كريميس تشامع باكو أفقاب صديق

[jongwook]

This is similar to #203, where the model unfortunately didn't learn to transcribe Somali at all.

For the cases like Hindi (#229) or Japanese (#204) where the model mostly recognizes the speech but outputs in a "wrong format", there is a chance to slightly nudge the model to output in a desired format. However, one would need to retrain the model to better support Somali or Bengali.

[maugas]

Thank you, I will look into it

[anubhav712]

I came across this notebook which fine-tunes the model with a Japanese corpus using PyTorch Lightning. I presume you can use a similar setup if you have a dataset in Somali.

hei excellent on notebook link thank you , i checked with the same corpus you attached , quick question

You said it fine tunes the model , what do you mean by it , next time we do not need to do anything ? it automatically picks up the correct translation

and which model will it correct

[jongwook]

Fine-tuning means reusing an existing model to quickly adapt to a new task by slightly changing the model weights. Resources like this might be helpful.

[swankwc]

I'm also getting Sinhala when I try to transcribe Somali. Even when I specify the language there are repeats of the same line. Some are correct and some are laughably bad. It would seem that fine-tuning is definitely in order. If any of you have had any luck with your efforts please let me know what you did to fix this.

[C0deXG]

Did you find way to do Somali speech to text if u do plz let me know I am facing the same issue thank u so much

[maugas]

No, I didn’t Mohamed On Oct 17, 2023, at 12:31 AM, COdXG ***@***.***> wrote:  Did you find way to do Somali speech to text if u do plz let me know I am facing the same issue thank u so much — Reply to this email directly, view it on GitHub<#234 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALQHXXAX6ZTS2LXKKFZBFMLX7YCZJAVCNFSM6AAAAAAQ3Y5FESVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TEOJZGYZDM>. You are receiving this because you authored the thread.Message ID: ***@***.***>