How to make Whisper understand my horrendous Australian accent?

[PatienceAllergy]

I use whisper (and previously Vosk) to help me pre-edit a podcast. My North American co-hosts/guests are transcribed quite well. But alas, me and my not-very-strong Aussie accent, often confuses the living word out of Whisper. Any suggestions?

[jschultz]

Have you gotten anywhere with this? I'm also looking for an open source transcription that can understand Australian English.

[glangford]

More training data with Australian-accented English would likely be required to improve accuracy. See

Fine-Tune Whisper For Multilingual ASR
https://huggingface.co/blog/fine-tune-whisper

The training data might use a dataset like AusTalk, for example - I don't know if the whisper authors would consider adding it in future or if you would need to create your own custom model.

A group of eleven major universities and 30 top speech science and technology experts from around Australia obtained funding under a Linkage Infrastructure, Equipment and Facilities (LIEF) grant from the Australian Government and a number of sponsors to create the largest-ever auditory-visual database of Australian speech. The project involved building twelve identical stand-alone recording set-ups and shipping these around Australia to collect audio-visual speech in each capital city and in various regional centres across the country (17 different collections in 15 locations).

https://austalk.edu.au

[camstuart]

I'm preparing a dataset for fine tuning based of Australian English (Victoria in particular).
It's a very specific domain (Firefighter radio comms), There are many Aussie speakers both male and female, and a lot of indigenous location names.

I would love to boost accuracy with more training data that has already been transcribed, but I wasn't able to find anything at that link mentioned above @glangford . If anyone has some data I can add to my training set please let me know.

Once I am finished I will be happy to share my model on hugging face.

[glangford]

The link I posted above isn't working for me now either. It looks like the AusTalk corpus may now be hosted here, but the links aren't working.
https://app.alveo.edu.au/catalog/austalk

For completeness, here is a description of the AusTalk research

http://www.lrec-conf.org/proceedings/lrec2014/pdf/520_Paper.pdf

[camstuart]

Thanks @glangford that’s very helpful

[WhiteyDude]

Looks like it's down again unfortunately.

@camstuart did you end up building that dataset? I've got hundreds of hours of Australian podcasts saved I was thinking manually editing so it's only voice and using whisper to transcribe to text and fine tune. Are you using the reference linked above, and how are you finding the results? Also, what hardware are you using and how long did the fine tuning take? Thanks!

[camstuart]

@WhiteyDude Not yet, I am collecting domain specific audio as we speak (a slow job, as I need to wait for radio comms to happen naturally) But I have definitely found that transcribing on longer snippets, and adding some leading silence has helped with default accuracy.

My plan is to then 'touch up' those transcriptions, and re train. hardware wise, I'll do this on google collab, and rent a decent GPU... thats my plan anyway. So far I probably only have about 2 hours worth. I am hoping to train on about 4 hours worth.

[s3m1s0n1c]

Any Movement on this one..

I too would love a Aussie Dataset for Whisper..

Thanks
Sonic

[glangford]

fyi in case this is helpful for anyone on this discussion thread

The Corpus of Australian and New Zealand Spoken English (CoANZSE) is a 196-million-word corpus of geolocated automatic speech recognition (ASR) YouTube transcripts from local government channels in Australia and New Zealand, created for the study of lexical, grammatical, and discourse-pragmatic phenomena of spoken language

https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi%3A10.7910%2FDVN%2FGW35AK

The Australian National Corpus is a national meta-collection that includes assorted examples of Australian English text (published and unpublished), transcriptions, audio and audio-visual materials from individual collections provided by collaborative institutions

https://researchdata.edu.au/australian-national-corpus/2018