TypeError: tokenizers.AddedToken() got multiple values for keyword argument 'special'

[umarbutler]

System Info

5.2.0

Who can help?

@ArthurZucker @itazap

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

My Hugging Face tokenizer no longer loads in Transformers v5. The tokenizer is isaacus/kanon-2-tokenizer. I am seeing this error when attempting to load it:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[1], line 3
      1 from transformers import AutoTokenizer
----> 3 tok = AutoTokenizer.from_pretrained("isaacus/kanon-2-tokenizer")

File ~/isaacus/cookbooks/.venv/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.py:712, in AutoTokenizer.from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    709     if tokenizer_class is None:
    710         tokenizer_class = TokenizersBackend
--> 712     return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    713 elif getattr(config, "tokenizer_class", None):
    714     _class = config.tokenizer_class

File ~/isaacus/cookbooks/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:1712, in PreTrainedTokenizerBase.from_pretrained(cls, pretrained_model_name_or_path, cache_dir, force_download, local_files_only, token, revision, trust_remote_code, *init_inputs, **kwargs)
   1709     if file_id not in resolved_vocab_files:
   1710         continue
-> 1712 return cls._from_pretrained(
   1713     resolved_vocab_files,
   1714     pretrained_model_name_or_path,
   1715     init_configuration,
   1716     *init_inputs,
   1717     token=token,
   1718     cache_dir=cache_dir,
   1719     local_files_only=local_files_only,
   1720     _commit_hash=commit_hash,
   1721     _is_local=is_local,
   1722     trust_remote_code=trust_remote_code,
   1723     **kwargs,
   1724 )

File ~/isaacus/cookbooks/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:1839, in PreTrainedTokenizerBase._from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, token, cache_dir, local_files_only, _commit_hash, _is_local, trust_remote_code, *init_inputs, **kwargs)
   1837     continue  # User-provided kwargs take precedence
   1838 if isinstance(value, dict) and key != "extra_special_tokens":
-> 1839     value = AddedToken(**value, special=True)
   1840 elif key == "extra_special_tokens" and isinstance(value, list):
   1841     # Merge list tokens, converting dicts to AddedToken
   1842     existing = list(init_kwargs.get("extra_special_tokens") or [])

TypeError: tokenizers.AddedToken() got multiple values for keyword argument 'special'

Expected behavior

The tokenizer should correctly load as it did previously.

[umarbutler]

I managed to fix this by deleting special_tokens_map.json from my tokenizer. It would seem a better idea, however, for transformers to simply ignore that file rather than bugging out like this...

[itazap]

I can't reproduce :( try pulling latest on main and let me know if you are still seeing the same error

[darshjme-codes]

Root Cause Analysis

This issue stems from a keyword argument collision in the AddedToken constructor within the tokenization loading pipeline. The error occurs at line 1839 in tokenization_utils_base.py where the code attempts:

value = AddedToken(**value, special=True)

Here's what's happening under the hood:

1. Dictionary unpacking collision: When value (a dictionary representing token configuration) contains a 'special' key, and we simultaneously pass special=True as a keyword argument, Python raises a TypeError because the same parameter is being passed twice.

2. API evolution mismatch: This suggests a breaking change between tokenizers library versions where the AddedToken constructor signature or the expected format of token configurations has changed.

3. Backward compatibility failure: The transformers library assumes that token configurations from older models will maintain compatibility with the current AddedToken instantiation logic, but this assumption breaks when the serialized token configuration already includes the special field.

The fundamental architectural flaw is in the blind application of default parameters without checking for existing keys in the configuration dictionary. This violates the principle of defensive programming and backward compatibility.

Three Solution Options (Ranked)

Option 1: Quick Fix (Recommended)

Handle the keyword collision by conditionally applying the special parameter only when not already present:

# In tokenization_utils_base.py around line 1839
if isinstance(value, dict) and key != "extra_special_tokens":
    # Create AddedToken while avoiding duplicate keyword arguments
    added_token_kwargs = value.copy()
    if 'special' not in added_token_kwargs:
        added_token_kwargs['special'] = True
    value = AddedToken(**added_token_kwargs)

Option 2: Proper Fix (Architectural)

Refactor the token configuration processing to use a more robust merging strategy that respects user intent while maintaining defaults:

def _create_added_token_from_config(token_config, default_special=True):
    """
    Safely create AddedToken from configuration dictionary,
    respecting existing special flag while providing sensible defaults.
    """
    if not isinstance(token_config, dict):
        raise TypeError(f"Expected dict for token config, got {type(token_config)}")
    
    # Copy config to avoid mutating original
    config = token_config.copy()
    
    # Handle special flag with proper precedence
    # If user explicitly set special=False, respect that
    # If user didn't specify, apply default
    if 'special' not in config:
        config['special'] = default_special
    
    # Validate required fields exist
    required_fields = {'content'}  # Assuming content is required
    missing_fields = required_fields - set(config.keys())
    if missing_fields:
        raise ValueError(f"Missing required fields in token config: {missing_fields}")
    
    return AddedToken(**config)

# Replace the problematic line with:
if isinstance(value, dict) and key != "extra_special_tokens":
    value = _create_added_token_from_config(value, default_special=True)

Option 3: Long-Term (Best Practice)

Implement a comprehensive token configuration schema validation system with version-aware deserialization:

from dataclasses import dataclass
from typing import Optional, Union
import json

@dataclass
class TokenConfigSchema:
    """Schema for validating and normalizing token configurations."""
    content: str
    special: bool = True
    single_word: bool = False
    lstrip: bool = False
    rstrip: bool = False
    normalized: Optional[bool] = None
    
    @classmethod
    def from_dict(cls, config_dict: dict, default_special: bool = True) -> 'TokenConfigSchema':
        """Safely construct from dictionary with proper defaults and validation."""
        # Start with defaults
        schema_dict = {
            'special': default_special,
            'single_word': False,
            'lstrip': False,
            'rstrip': False
        }
        
        # Update with user-provided values
        schema_dict.update({k: v for k, v in config_dict.items() 
                           if hasattr(cls, k) or k in ['content', 'normalized']})
        
        # Validate content exists
        if 'content' not in schema_dict:
            raise ValueError("Token configuration must include 'content' field")
            
        return cls(**schema_dict)
    
    def to_added_token(self) -> 'AddedToken':
        """Convert to AddedToken instance."""
        return AddedToken(**self.__dict__)

# Updated processing logic:
if isinstance(value, dict) and key != "extra_special_tokens":
    token_schema = TokenConfigSchema.from_dict(value, default_special=True)
    value = token_schema.to_added_token()

Production Test Code

import unittest
from unittest.mock import Mock, patch
from transformers.tokenization_utils_base import AddedToken
from transformers.models.auto.tokenization_auto import AutoTokenizer
import tempfile
import os
import json


class TestAddedTokenCollisionFix(unittest.TestCase):
    
    def setUp(self):
        """Set up test fixtures before each test method."""
        # Mock the problematic scenario
        self.token_config_with_special = {
            'content': '<pad>',
            'special': True,
            'single_word': False
        }
        
        self.token_config_without_special = {
            'content': '<unk>',
            'single_word': False
        }
        
        self.token_config_explicit_false = {
            'content': 'custom_token',
            'special': False,
            'lstrip': True
        }
    
    def test_original_bug_reproduction(self):
        """Test that demonstrates the original bug."""
        # This would fail with: TypeError: tokenizers.AddedToken() got multiple values for keyword argument 'special'
        with self.assertRaises(TypeError):
            # Simulate the original problematic code
            config = self.token_config_with_special.copy()
            # This mimics: AddedToken(**value, special=True)
            AddedToken(**config, special=True)
    
    def test_quick_fix_solution(self):
        """Test the quick fix solution."""
        def safe_create_added_token(token_config, default_special=True):
            added_token_kwargs = token_config.copy()
            if 'special' not in added_token_kwargs:
                added_token_kwargs['special'] = default_special
            return AddedToken(**added_token_kwargs)
        
        # Test case 1: Config already has special=True
        token1 = safe_create_added_token(self.token_config_with_special)
        self.assertTrue(token1.special)
        self.assertEqual(str(token1), '<pad>')
        
        # Test case 2: Config doesn't have special (should get default)
        token2 = safe_create_added_token(self.token_config_without_special)
        self.assertTrue(token2.special)  # Should get default True
        
        # Test case 3: Config has special=False (should respect user choice)
        token3 = safe_create_added_token(self.token_config_explicit_false)
        self.assertFalse(token3.special)
    
    def test_proper_fix_solution(self):
        """Test the proper fix solution."""
        def _create_added_token_from_config(token_config, default_special=True):
            if not isinstance(token_config, dict):
                raise TypeError(f"Expected dict for token config, got {type(token_config)}")
            
            config = token_config.copy()
            if 'special' not in config:
                config['special'] = default_special
            
            required_fields = {'content'}
            missing_fields = required_fields - set(config.keys())
            if missing_fields:
                raise ValueError(f"Missing required fields in token config: {missing_fields}")
            
            return AddedToken(**config)
        
        # All test cases should work
        token1 = _create_added_token_from_config(self.token_config_with_special)
        token2 = _create_added_token_from_config(self.token_config_without_special)
        token3 = _create_added_token_from_config(self.token_config_explicit_false)
        
        self.assertTrue(token1.special)
        self.assertTrue(token2.special)
        self.assertFalse(token3.special)
    
    def test_edge_cases(self):
        """Test edge cases and error conditions."""
        def safe_create_added_token(token_config, default_special=True):
            added_token_kwargs = token_config.copy()
            if 'special' not in added_token_kwargs:
                added_token_kwargs['special'] = default_special
            return AddedToken(**added_token_kwargs)
        
        # Test empty config (should fail due to missing content)
        with self.assertRaises(Exception):  # AddedToken likely requires content
            safe_create_added_token({})
        
        # Test config with all possible fields
        full_config = {
            'content': '[MASK]',
            'special': True,
            'single_word': True,
            'lstrip': False,
            'rstrip': True,
            'normalized': True
        }
        token = safe_create_added_token(full_config)
        self.assertTrue(token.special)
        self.assertTrue(token.single_word)
    
    def test_backward_compatibility(self):
        """Test that old token configurations still work."""
        # Simulate old-style token configs that might have been saved
        old_configs = [
            {'content': '<pad>'},  # No special field
            {'content': '<bos>', 'special': True},  # Explicit special=True
            {'content': '<eos>', 'special': False},  # Explicit special=False
            {'content': '<sep>', 'single_word': True},  # Mixed with other params
        ]
        
        def safe_create_added_token(token_config, default_special=True):
            added_token_kwargs = token_config.copy()
            if 'special' not in added_token_kwargs:
                added_token_kwargs['special'] = default_special
            return AddedToken(**added_token_kwargs)
        
        for i, config in enumerate(old_configs):
            token = safe_create_added_token(config)
            if 'special' in config:
                self.assertEqual(token.special, config['special'], 
                               f"Config {i} should preserve explicit special value")
            else:
                self.assertEqual(token.special, True, 
                               f"Config {i} should get default special value")


def create_mock_tokenizer_files():
    """Create temporary files that reproduce the original issue."""
    temp_dir = tempfile.mkdtemp()
    
    # Create a mock tokenizer.json that would trigger the bug
    tokenizer_config = {
        "model": {
            "type": "BPE",
            "vocab": {"<pad>": 0, "<unk>": 1},
            "merges": []
        },
        "added_tokens": [
            {"id": 0, "content": "<pad>", "special": True, "single_word": False}
        ],
        "normalizer": None,
        "pre_tokenizer": None,
        "post_processor": None,
        "decoder": None
    }
    
    config_file = os.path.join(temp_dir, "tokenizer.json")
    with open(config_file, 'w') as f:
        json.dump(tokenizer_config, f)
    
    # Create config.json
    config_content = {
        "model_type": "gpt2",  # or whatever model type
        "tokenizer_class": "GPT2Tokenizer"
    }
    config_path = os.path.join(temp_dir, "config.json")
    with open(config_path, 'w') as f:
        json.dump(config_content, f)
    
    return temp_dir


if __name__ == "__main__":
    # Run the tests
    unittest.main(verbosity=2)
    
    # Example usage of the fix
    print("\n=== Demonstrating the fix ===")
    
    # Original problematic code (would fail):
    # config = {'content': '<pad>', 'special': True}
    # AddedToken(**config, special=True)  # TypeError!
    
    # Fixed approach:
    config = {'content': '<pad>', 'special': True}
    added_token_kwargs = config.copy()
    if 'special' not in added_token_kwargs:
        added_token_kwargs['special'] = True
    token = AddedToken(**added_token_kwargs)
    print(f"Successfully created token: {token}")
    print(f"Token special flag: {token.special}")

Impact Assessment

• Severity: HIGH - This is a breaking change that prevents loading of previously valid tokenizers, affecting any user with custom tokenizers that include explicit special flags in their configuration.

• Affected users: Anyone using custom tokenizers (like isaacus/kanon-2-tokenizer) that were created with older versions of the transformers library and contain explicit special field specifications in their added tokens configuration.

• Consequences: Users cannot upgrade to Transformers v5 without either recreating their tokenizers or manually fixing the configuration files. This creates friction in the ecosystem and breaks existing workflows.

• Migration path: The fix is backward compatible - existing tokenizers without explicit special flags will continue to work as before, while those with explicit flags will now be handled correctly instead of throwing errors.

---

Built by @darshjme-codes — Creator of Samast (universal LLM provider abstraction) and Brahmand CLI.