All tokenizers raise incorrect regex pattern warning after version 4.57.3?

[jue-jue-zi]

transformers/src/transformers/tokenization_utils_base.py

Line 2466 in 753d611

if transformers_version and version.parse(transformers_version) <= version.parse("4.57.2"):

The #42299 pr make all tokenizers raise incorrect regex pattern warning after version 4.57.3, such as qwen model type, is it correct?

[Rocketknight1]

Hi @jue-jue-zi, can you give us some example code that triggers the warning?

[jue-jue-zi]

# transformers==4.57.6
from transformers import AutoConfig
from transformers import AutoModel
from transformers import AutoTokenizer
from transformers import GenerationConfig

pretrain_path = "Qwen/Qwen3-0.6B"
local_path = "/tmp/Qwen3-0.6B"

tokenizer = AutoTokenizer.from_pretrained(pretrain_path)
model_config = AutoConfig.from_pretrained(pretrain_path)
model = AutoModel.from_config(model_config)
gen_config = GenerationConfig.from_pretrained(pretrain_path)

tokenizer.save_pretrained(local_path)
model.save_pretrained(local_path)
model.save_pretrained(local_path)
gen_config.save_pretrained(local_path)


t = AutoTokenizer.from_pretrained(local_path)  # <- The tokenizer you are loading from '/tmp/Qwen3-0.6B' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.

Thanks for reply, a simple code is shown as above @Rocketknight1.

[Rocketknight1]

Hi @jue-jue-zi, I just tried with the latest version from main and I didn't see the warning! Can you try updating and see if that fixes the bug?

[jue-jue-zi]

But 4.57.6 is the latest version of transformers v4? @Rocketknight1

[Rocketknight1]

Hi @jue-jue-zi, we're not making any further fixes to the v4 branch! The current version is v5.