Bug of decoder max_length

[ILG2021]

Recently I found faster whisper will output not complete text and cut off the last char, and I have tried to locate the bug. When I use transformers to inference, it is ok. But it occurs on faster whisper and openai whisper. So I dig into the code, and I found openai whisper has a line of code to limit the decode output length to 224:

self.sample_len: int = options.sample_len or model.dims.n_text_ctx // 2

It is on line 529 of decoding.py of project openai whisper. When I change it to:

self.sample_len: int = options.sample_len or model.dims.n_text_ctx

everything it is ok.

[janniss91]

Hey!

Did you also find a solution for faster_whisper when this happens?

Or only for openai/whisper?

[ILG2021]

faster whisper has the same problem, i have rebuild ctranslate2 to solve it.

[20IM30007]

Hey @ILG2021, can we do the same thing if we also use prompt/condition_on_previous_text = True? From my analysis of both paper and code, I understand that 224 is preserved for prompt and 224 is reserved for output. But can vary if the prompt given is less than 224 tokens.

[20IM30007]

def _get_initial_tokens(self) -> Tuple[int]:
tokens = list(self.sot_sequence)

    if prefix := self.options.prefix:
        prefix_tokens = (
            self.tokenizer.encode(" " + prefix.strip())
            if isinstance(prefix, str)
            else prefix
        )
        if self.sample_len is not None:
            max_prefix_len = self.n_ctx // 2 - self.sample_len
            prefix_tokens = prefix_tokens[-max_prefix_len:]
        tokens = tokens + prefix_tokens

    if prompt := self.options.prompt:
        prompt_tokens = (
            self.tokenizer.encode(" " + prompt.strip())
            if isinstance(prompt, str)
            else prompt
        )
        tokens = (
            [self.tokenizer.sot_prev]
            + prompt_tokens[-(self.n_ctx // 2 - 1) :]
            + tokens
        )

    return tuple(tokens)

can we just limit the prompt_tokens length to 100 like in prompt_tokens[-100 :] and then use custom value of sample_len to achieve this? @jongwook

[ILG2021]

refix_tokens = (
self.tokenizer.encode(" " + prefix.strip())
if isinstance(prefix, str)
else prefix
)
if self.sample_len is not None:
max_prefix_len = self.n_ctx // 2 - self.sample_len
prefix_tokens = prefix_tokens[-max_prefix_len:]
tokens = tokens + prefix_tokens

if prompt := s

I don't know if it can work. you can have a try. I know that the implement of huggingface transformers library don't have this problem, you can dig it. it may contains many redundancy code because it is a common use library.

my change in ctranslte2 is here:
OpenNMT/CTranslate2@8423d54#diff-3ff4ae15a49bfc50b7a62e9fcb5c5fa380b53f8840fa4e5eabbc2c64947b9d18

I have write how to compile in readme.