TOKENIZER_MAPPING_NAMES sometimes returns None, but from_pretrained assumes otherwise

[DavidMChan]

System Info

When loading a tokenizer with AutoTokenizer (src/transformers/models/auto/tokenization_auto.py), on L652 in from_pretained the code tries to remove "Fast" from the tokenizer mapping for old-style models:

if (
            tokenizer_auto_map is None
            and tokenizer_config_class is not None
            and config_model_type is not None
            and config_model_type != ""
            # Here:
            and TOKENIZER_MAPPING_NAMES.get(config_model_type, "").replace("Fast", "")
            != tokenizer_config_class.replace("Fast", "")
        ):

Permalink:

transformers/src/transformers/models/auto/tokenization_auto.py

Line 652 in ffbea2d

and TOKENIZER_MAPPING_NAMES.get(config_model_type, "").replace("Fast", "")

However, sometimes TOKENIZER_MAPPING_NAMES.get(config_model_type, "") returns None, instead of an empty string (if the config_model_type is in the dict, but has value None), leading to models failing to load with an attribute error

AttributeError: 'NoneType' object has no attribute 'replace'

This could be fixed as follows:

and (TOKENIZER_MAPPING_NAMES.get(config_model_type, "") or "")

Or by some other reasonable check.

Who can help?

Looks like this was introduced in #42894 - @ArthurZucker and @itazap

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Run:

from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained('google/siglip2-so400m-patch14-384')

Expected behavior

It shouldn't throw an attribute error :)

[Rocketknight1]

Fixed in #44127 I think! Feel free to ping me or reopen if the issue is still there after you install from main.

[jkj-dev-rg]

@Rocketknight1
Im also getting the same error now, currently im using the latest vesion 5.2
PS C:\Users\jyoth> docker exec -it vibrant_jennings pip show transformers
Name: transformers
Version: 5.2.0
Summary: Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
Home-page: https://github.com/huggingface/transformers
Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors)
Author-email: transformers@huggingface.co
License: Apache 2.0 License
Location: /opt/venv/lib/python3.10/site-packages
Requires: huggingface-hub, numpy, packaging, pyyaml, regex, safetensors, tokenizers, tqdm, typer-slim
Required-by:

Error trace back:

{"status":"error","message":"Full traceback:

Traceback (most recent call last):
File "/app/src/modules/embedding/siglip_embedding_server.py", line 368, in load_model
model, processor, image_processor, tokenizer, device =
load_siglip_robust(dest_dir, EXPECTED_DIM, local_files_only=True)

File "/app/src/modules/embedding/siglip_embedding_server.py", line 103, in load_siglip_robust
_cached_processor = AutoProcessor.from_pretrained(model_name_or_path, **processor_kwargs)

File "/opt/venv/lib/python3.10/site-packages/transformers/models/auto/processing_auto.py", line 402, in from_pretrained
return processor_class.from_pretrained(...)

File "/opt/venv/lib/python3.10/site-packages/transformers/processing_utils.py", line 1400, in from_pretrained
args = cls._get_arguments_from_pretrained(...)

File "/opt/venv/lib/python3.10/site-packages/transformers/processing_utils.py", line 1514, in _get_arguments_from_pretrained
tokenizer = cls._load_tokenizer_from_pretrained(...)

File "/opt/venv/lib/python3.10/site-packages/transformers/processing_utils.py", line 1461, in _load_tokenizer_from_pretrained
tokenizer = auto_processor_class.from_pretrained(...)

File "/opt/venv/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 652, in from_pretrained
and TOKENIZER_MAPPING_NAMES.get(config_model_type, "").replace("Fast", "")
AttributeError: 'NoneType' object has no attribute 'replace'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/app/src/modules/embedding/siglip_embedding_server.py", line 103, in load_siglip_robust
_cached_processor = AutoProcessor.from_pretrained(model_name_or_path, **processor_kwargs)

(same Transformers call chain as above...)

File "/opt/venv/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 652, in from_pretrained
and TOKENIZER_MAPPING_NAMES.get(config_model_type, "").replace("Fast", "")
AttributeError: 'NoneType' object has no attribute 'replace'
"}

same code base run in local but not in docker, both local and docker have same transformers version,
in local model artifacts were provided after downloading from hf siglip2-giant-opt-patch16-384, same model were mounted -volume to docker to load the model.

[ArthurZucker]

we'll release to fix this

[jkj-dev-rg]

I pulled the main from git and installed the package. Now its working fine.
BTW May I know the reason behind the issue. It worked in local environment with same package versions but not in docker build environment. @ArthurZucker

[IDoCodingStuffs]

AutoProcessor is broken until this lands (below fails on transformers>=5.1.0, works on transformers==4.57.6):

model_id = "google/siglip2-large-patch16-512"
processor = AutoProcessor.from_pretrained(model_id,)