Improve quality of language support?

[AntiDotZA]

As part of my job (both day and night) I often need to create English subtitles from Afrikaans TV shows. For this I use the opensource Subtitle Edit application. Very handy.
However, the subtitling process is tedious and doing this manually takes a ridiculous amount of time. (For me, somewhere along the lines of 30minutes per content minute.)
After some research online regarding transcription software, I found out that Subtitle Edit has integrated Whisper for an "Audio to Text" feature, and unlike other software options, Whisper has language support for Afrikaans.
So I spent the last day getting this feature to work (it involves running a lot of coded command lines outside of my comfort zone), and FINALLY it's working. With much anticipation I ran the transcription feature on an Afrikaans video to see check the accuracy....
It was a complete dud. The results resembled Dutch more than Afrikaans, and is not usable at all. :(
I want to cry! Afrikaans is my home language, and even when it does get some representation in the software world, it is still confused with something that is not recognisable as Afrikaans. (Please see attached a few lines from the automatic transcription vs a manual transcription.)
So my question:
How can I help to improve the transcription results on Afrikaans? Because there are others that do the same work as me, and we will all benefit from a more accurate interpretation of spoken Afrikaans.
Many thanks!!!

[glangford]

From this link, you can see that whisper has a relatively high error rate for Afrikaans. This is because Whisper has not been trained on enough Afrikaans audio yet.
https://raw.githubusercontent.com/openai/whisper/main/language-breakdown.svg

People are working on fine tuning Whisper for different languages, but Afrikaans has not been targeted yet. See for example

• Can I train Whisper in my language ? #276

In the short term, since you are not a developer, you can:

• make sure whisper really is detecting Afrikaans as the language and is not detecting Dutch in error; you can force the language using --language af on the command line for whisper
• if you know certain words that will appear in the transcript, you can give hints to whisper by using --initial_prompt; see How to make Whisper recognise more words #288
• use the "large" model in transcription (slower but generally more accurate) using --model large. It may be too large or too slow for your needs however.
• experiment with other command line options to find what works best. For my (non-English) language that I transcribe most, I find --patience 2 is helpful for example

[glangford]

p.s. Since you are creating English subtitles, note that whisper can translate (to English) as well as transcribe. I see from the screenshot that you are showing transcription. I don't know if/how well this is integrated into Subtitle Edit, but you can specify Afrikaans as the source language and translate to English using --language af and --task translate

[AntiDotZA]

Thank you for getting back to me.

I saw that the Afrikaans interpretation is not flawless, but I didn't expect it to turn out like a completely different language.
The "large" model threw a "No text found!" error, which is why I'm running "base" at the moment, and I can't figure out how to get hold of large-v2.

A thread that let me lift my ears were this one: #13
Since I do have some programming knowledge (granted, only C# and I haven't done any programming in 7 years), I tried to execute these steps in Google Colab:
https://colab.research.google.com/github/sanchit-gandhi/notebooks/blob/main/fine_tune_whisper.ipynb#scrollTo=f12e2e57-156f-417b-8cfb-69221cc198e8
I got on great, until I got an error that Afrikaans is not in the Common Voice V11.0 data set, so I can't do any training for it. :/
Maybe Charlize Theron is not the only person that thinks only 44 people in the world speaks Afrikaans... ;)

Any other suggestions will be great!

[AntiDotZA]

p.s. Since you are creating English subtitles, note that whisper can translate (to English) as well as transcribe. I see from the screenshot that you are showing transcription. I don't know if/how well this is integrated into Subtitle Edit, but you can specify Afrikaans as the source language and translate to English using --language af and --task translate

I think that since the original audio is misinterpreted as Dutch and therefore nonsensical at times, the English translation won't be great. :/

[glangford]

The "large" model threw a "No text found!" error, which is why I'm running "base" at the moment, and I can't figure out how to get hold of large-v2.

For now I would try the small or medium model and forcing with --language. You can get large-v2 by upgrading to the latest whisper, and after that specifying --model large implies large-v2.

I think that since the original audio is misinterpreted as Dutch and therefore nonsensical at times, the English translation won't be great. :/

But forcing it to recognize Afrikaans and translating is worth the experiment ie. --language af --task translate

[Code4SAFrankie]

This might be of interest as well, although I have no idea how to use it with whisper: https://huggingface.co/Ari/whisper-base-af-za

[dgoryeo]

Hi @AntiDotZA , regarding the "No Text found" error, is that error coming from SubtitleEdit/WhisperCPP, or from the model itself?

I used to get "No Text found" error when trying to transcribe from Japanese using SubtitleEdit. At the time I concluded that the error comes from SubtitleEdit implementation/integration of WhisperCPP, but I might be wrong of course. The community in both SE and WhsiperCPP are quite helpful and might know some workaround.

[AntiDotZA]

Hey! I found this thorough video on resolving the issue: https://www.youtube.com/watch?v=Al894WeQmgU
Currently, I don't have any issues with running the "base" or "medium" models, only "large". This indicates to me that the problem might be isolated to the "large" model.
I tried to figure out where the error came from by investigating the errors in the Windows Event Viewer, but beyond upgrading, I didn't learn a lot.
At the moment, I'm happy to carry on with the medium model, but the accuracy on Afrikaans must be improved. I'm happy to assist with that process as a first language speaker with a bit of tech savvy, if I can just figure out how to do that...