Tokenization related error on some machines

[Mindful]

Hey all, for whatever reason I can't load the model on my laptop. It loads fine on a GPU server and I can run it there, but any attempt to load it on my laptop gives me an error that looks like the below.

This is with a fresh Conda environment, so I'm not entirely sure what the issue could be. I also updated ffmpeg, so I'm not sure what the issue is. Any thoughts or possible solutions would be great; life would be much easier if I could run this locally too. Thanks!

Command I ran: whisper ~/Downloads/test.mp3 --model small
Error (warning is included just in case it might be relevant).

/home/josh/miniconda3/envs/whisper/lib/python3.9/site-packages/whisper/transcribe.py:70: UserWarning: FP16 is not supported on CPU; using FP32 instead
  warnings.warn("FP16 is not supported on CPU; using FP32 instead")
Detecting language using up to the first 30 seconds. Use `--language` to specify the language
Traceback (most recent call last):
  File "/home/josh/miniconda3/envs/whisper/bin/whisper", line 8, in <module>
    sys.exit(cli())
  File "/home/josh/miniconda3/envs/whisper/lib/python3.9/site-packages/whisper/transcribe.py", line 279, in cli
    result = transcribe(
  File "/home/josh/miniconda3/envs/whisper/lib/python3.9/site-packages/whisper/transcribe.py", line 82, in transcribe
    _, probs = model.detect_language(segment)
  File "/home/josh/miniconda3/envs/whisper/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/josh/miniconda3/envs/whisper/lib/python3.9/site-packages/whisper/decoding.py", line 33, in detect_language
    tokenizer = get_tokenizer(model.is_multilingual)
  File "/home/josh/miniconda3/envs/whisper/lib/python3.9/site-packages/whisper/tokenizer.py", line 318, in get_tokenizer
    tokenizer = build_tokenizer(name=tokenizer_name)
  File "/home/josh/miniconda3/envs/whisper/lib/python3.9/site-packages/whisper/tokenizer.py", line 277, in build_tokenizer
    tokenizer = GPT2TokenizerFast.from_pretrained(path)
  File "/home/josh/miniconda3/envs/whisper/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1775, in from_pretrained
    return cls._from_pretrained(
  File "/home/josh/miniconda3/envs/whisper/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1805, in _from_pretrained
    slow_tokenizer = (cls.slow_tokenizer_class)._from_pretrained(
  File "/home/josh/miniconda3/envs/whisper/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1986, in _from_pretrained
    raise ValueError(
ValueError: Non-consecutive added token '<|endoftext|>' found. Should have index 93 but has index 50257 in saved vocabulary.

[r0y6a3n0]

Hi, I got same problem. l solved this problem by downgrading transformers version to 4.22.0. Hope this helps.

[Mindful]

I'll try this when I get home - thanks!

[Mindful]

Yup, this solved it.
@r0y6a3n0 thank you!