Higher beam size causes Whisper to skip transcribing certain parts. Why?

[FurkanGozukara]

I have just tested 15 beam_size.

It transcribed almost perfect the parts it didn't fail.

After like 4 minutes there is like 1 minute speech it failed to transcribe.

When beam_size reduced to 5 it transcribed all. Do you know why? How to find a sweet spot?

[hoonlight]

You can refer to this paper

Many researchers observe that translation quality degrades with beam sizes beyond 5 or 10 (Tuet al., 2017; Koehn and Knowles, 2017). We call this phenomenon the “beam search curse”, which is listed as one of the six biggest challenges for NMT (Koehn and Knowles, 2017).

[glangford]

How to find a sweet spot?

Trial and error until you find the best settings for your language and type of content. I use beam_size 5 and patience 2 with the large model (most of the time) for the Portuguese language for example, with excellent results.