How to personalize/finetune Whisper dynamically by user in application

[albcunha]

Hello!
I have different users on my application I want to allow them to add word that are mistranscribed by whisper. One common situation is the user name. For example, Aaron and Erin are too similar. I think that a specific approach by user would be better, as there are also some techinal words they say and because they are talking to other techinical people, they tend to speak it very fast and dont fully pronounce it.

I was wondering if a adapter approach https://huggingface.co/docs/hub/en/adapters would work on this case and if there are any suggestion as how to tackle this.

I´m sorry if if it´s a simple question but a searched for adapter on the discussion and I only found people trying to create new languages on the model.

[glangford]

Have you tried using --initial_prompt with the user's correctly spelled name and a personalized preamble?

See

• How can I teach whisper new words? #963
• Suggesting Vocab for higher accuracy #328

[albcunha]

Yes. It didn´t work. As some words are technical the model may not know them or it´s because they are mispronounced. Do you think the adapter strategy might work?

[glangford]

I don't have any insight into the adapter approach.

[glangford]

No affiliation with the company but saw this news item today. FYI.

... The AdaKWS model uses a novel keyword spotting method that integrates seamlessly into business workflows, allowing for the automation of reports and inspections via spoken commands. It is a machine learning algorithm that has been trained to augment another existing speech-to-text model — say, OpenAI’s Whisper or any other model of the customer’s choosing — fitting between the host model’s encoder, the part that listens to the speaker’s voice and words, and the decoder, which turns the audio into the correct text output.

Powerful new AI model accurately converts speech to text, even your company’s jargon
https://venturebeat.com/ai/exclusive-powerful-new-ai-model-accurately-converts-speech-to-text-even-your-companys-jargon/