Handling special tokens in initial_prompt

[Thytu]

Context

I currently don't see any way of handling special tokens, neither for allowing or disabling.
Is there a way to provide either allowed_special or disallowed_special parameters when encoding the initial_prompt?

If yes, how? If no, is that intended?

To reproduce

Code:

import whisper

model = whisper.load_model("base") # works with every models

result = model.transcribe("some-path.wav", initial_prompt="<|transcribe|>")

Output:

Encountered text corresponding to disallowed special token '<|{SOME-SPECIAL-TOEKN}|>'. If you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|transcribe|>', ...}`. If you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|transcribe|>'})`. To disable this check for all special tokens, pass `disallowed_special=()`.

dummy fix

Change:

if initial_prompt is not None:
-       initial_prompt_tokens = tokenizer.encode(" " + initial_prompt.strip())
+       initial_prompt_tokens = tokenizer.encode(
+           text=" " + initial_prompt.strip(),
+           allowed_special=decode_options.pop("allowed_special", set()),
+           disallowed_special=decode_options.pop("disallowed_special", "all"),
+       )
        all_tokens.extend(initial_prompt_tokens)
    else:
        initial_prompt_tokens = []

I insist on "dummy", ofc it needs to be done properly.

[maccman]

Also encountering this.