On Fine tuning , how to mark 'unknown' word , and 'noises' and silence ?

[itaipee]

I'm working to fine-tune Whisper models to my domain , and i was wondering :
if there unknown words in the manuals , in my manuals there are marked by question mark how do i mark them for the fin tuning ?
Same question for noises ( and if there are types of noises or everything is labeled the same?

another less common is silence . let says the 30 seconds audio contains 10 seconds of silence , and only then 20 seconds of speech.
How can i mark the silence ?

[phineas-pta]

u can't

[dgoryeo]

@itaipee , @phineas-pta , would this approach be an option for the question: Whisper-AT ?

[phineas-pta]

idk if it can help reduce hallucinations but it's worth a try

[itaipee]

interesting work is whisper-AT , thank you @dgoryeo for referring me .

my question is more simple , I want to fine-tune whisper with my data . the manual transcribed text contains labels for noise ,and unknown words ( tagger could not understand) , so what are the convention for tagging such acoustic events in current Whisper ? unknown words can be seems on different corpora as ~VN , (?), [unknown]

[phineas-pta]

whisper doesn't tag these events nor words

[dgoryeo]

@itaipee , I second the reply from @phineas-pta. In other words your problem description falls under the umbrella of classification, that's why Whipser-AT has extended/modified standard Whisper to produce "tagging" results.

[itaipee]

@itaipee , @phineas-pta , would this approach be an option for the question: Whisper-AT ?