Wrong tokens probabilities

[Rtut654]

Hello. I used whisper version from this PR, only difference - I changed weights to "small". I'm facing the following issue:
I made distorted speech example https://drive.google.com/file/d/12zGWllJg6edftcnwuHX_ZHMuwk7PlVjg/view?usp=sharing .
If I don't set the language of decoding i.e. options = whisper.DecodingOptions() , the output is correct in terms of locating mispronounce (I can read this Ukrainian) though it translates it to random language.

But if I set 'en' for decoding options = whisper.DecodingOptions(language="en") the picture and probabilities are wrong.
.
I checked a few similar audios and result is almost same. Probs are high for each word except the last one.

[phineas-pta]

this feature isnt official yet, u have to ask its author

[phineas-pta]

did u use original whisper ? it's same results ?

[Rtut654]

Yes, the result is the same, I just added a few lines to get token_probs :

 def update(
        self, tokens: Tensor, logits: Tensor, sum_logprobs: Tensor, token_probs: Tensor
    ) -> Tuple[Tensor, bool]:
        if self.temperature == 0:
            next_tokens = logits.argmax(dim=-1)
        else:
            next_tokens = Categorical(logits=logits / self.temperature).sample()

        logprobs = F.log_softmax(logits.float(), dim=-1)
        probs = torch.exp(logprobs)
        
        current_logprobs = logprobs[torch.arange(logprobs.shape[0]), next_tokens]

        next_tokens[tokens[:, -1] == self.eot] = self.eot
        tokens = torch.cat([tokens, next_tokens[:, None]], dim=-1)

        current_token_probs = probs[torch.arange(probs.shape[0]), next_tokens]
        token_probs = torch.cat([token_probs, current_token_probs[:, None]], dim=-1)
        
        completed = (tokens[:, -1] == self.eot).all()
        return tokens, completed, token_probs

[phineas-pta]

wait a sec, u set language="en" should have used task="translate"

[Rtut654]

Haven't got why, I want to transcribe audio in english to english text. Isn't it 'transcribe' task?

[phineas-pta]

nvm, i thought the audio is ukrainian not english

the quality is so bad that whisper misdetect the language, did u try better quality audio that whisper detect language correctly ?