I want to train proper nouns about industry

[Choi-YoungHyun]

Hi

I'm studying Speech to text about Korean.

Whisper is good performance. So I try to fine-tuning about proper nouns in Korean.

So, I made 1000 short sentences. (male is man voice / 500 female voice) and I conducted fine-tuning, but the proper nouns I trained are not being recognized well. I'd like to know how many examples are needed to train a single word effectively.

Model: small

[glangford]

FYI, related news coverage on a company called aiOla and their published research in case it is helpful (I have no affiliation with them)

Speech recognition AI learns industry jargon with aiOla’s novel approach

https://venturebeat.com/ai/exclusive-speech-recognition-ai-learns-industry-jargon-with-aiolas-novel-approach/

Keyword-Guided Adaptation of Automatic Speech Recognition

... In this paper, we propose a novel approach for improved jargon word recognition by contextual biasing Whisper-based models. We employ a keyword spotting model that leverages the Whisper encoder representation to dynamically generate prompts for guiding the decoder during the transcription process.

https://arxiv.org/pdf/2406.02649

[rahulbansal16]

Is it possible to fine tune with particular voice.

My product allows people to talk, and once they see the transcription, they correct it.
I want to train the whisper so that it gives better output for that particular user's voice.
Basically a fine tuned model for each user's voice and slowly it gets better.
How can I do that?