Why does Whisper become less functional the longer that it's used?

[ehobson]

This has happened to me across numerous devices. Whisper becomes less and less operational the longer that it is in use. Translations and transcriptions become less accurate. Hallucinations increase. It stops segmenting speech. Even if the same file is processed over and over, Whisper will give different results each time, with the results getting progressively worse.

This is a very annoying problem and any suggestions on a resolution would be appreciated. Closing the task window and restarting Whisper does not resolve the problem.

[gustavostz]

Hey there!

First of all, I'd like to mention that my experience with Whisper has been consistently good over time.

But let's try to address your issue:

1. Model and Language: Could you please share which model you are using and the language you are transcribing and translating from?

2. Varying Results: The issue of getting different results for the same file over time could be due to the temperature parameter in the transcribe function, which controls the randomness in the sampling process. If temperature is set as a tuple, Whisper will use different temperature values upon encountering errors, depending on either compression_ratio_threshold or logprob_threshold, leading to variations in the output. To ensure more consistent results, consider setting temperature to a single low value (e.g., 0.0) to reduce randomness. Additionally, make sure that condition_on_previous_text is set to True to maintain consistency across windows, unless you are experiencing issues with the model getting stuck in a failure loop. Also, use the same initial_prompt for each run if applicable.

I hope this helps! If the issue persists, please provide more details, and I'll try to help you further.

By the way, just in case, here is all the params config that you can use:

Parameters
    ----------
    model: Whisper
        The Whisper model instance

    audio: Union[str, np.ndarray, torch.Tensor]
        The path to the audio file to open, or the audio waveform

    verbose: bool
        Whether to display the text being decoded to the console. If True, displays all the details,
        If False, displays minimal details. If None, does not display anything

    temperature: Union[float, Tuple[float, ...]]
        Temperature for sampling. It can be a tuple of temperatures, which will be successively used
        upon failures according to either `compression_ratio_threshold` or `logprob_threshold`.

    compression_ratio_threshold: float
        If the gzip compression ratio is above this value, treat as failed

    logprob_threshold: float
        If the average log probability over sampled tokens is below this value, treat as failed

    no_speech_threshold: float
        If the no_speech probability is higher than this value AND the average log probability
        over sampled tokens is below `logprob_threshold`, consider the segment as silent

    condition_on_previous_text: bool
        if True, the previous output of the model is provided as a prompt for the next window;
        disabling may make the text inconsistent across windows, but the model becomes less prone to
        getting stuck in a failure loop, such as repetition looping or timestamps going out of sync.

    word_timestamps: bool
        Extract word-level timestamps using the cross-attention pattern and dynamic time warping,
        and include the timestamps for each word in each segment.

    prepend_punctuations: str
        If word_timestamps is True, merge these punctuation symbols with the next word

    append_punctuations: str
        If word_timestamps is True, merge these punctuation symbols with the previous word

    initial_prompt: Optional[str]
        Optional text to provide as a prompt for the first window. This can be used to provide, or
        "prompt-engineer" a context for transcription, e.g. custom vocabularies or proper nouns
        to make it more likely to predict those word correctly.

    decode_options: dict
        Keyword arguments to construct `DecodingOptions` instances

[bullhorn-admin]

I experienced the same issue. This is partly why I am switching to 5 min segments.