Tokenizers not model-agnostic

[urroxyz]

I believe that the library's support models are based on tokenization and not model brand.

For example, using any of the latest Falcon models should work because they run on PreTrainedTokenizerFast, however, the following error arises:

NotImplementedError: Tokenizer not supported: PreTrainedTokenizerFast

This seems to be because the library recognizes models based on their path or title as opposed to their tokenizer type, and so tiiuae/Falcon3-1B-Base is unsupported, even though it should not be. The solution is to compare the loaded tokenizer to a list of supported tokenizer classes instead of focusing on naming conventions.

[Saibo-creator]

That's correct. The model path-based check is here:
TokenizerMiddleMapping.py, line 35.

In the earlier versions of TCFG, I used a model class-based check, but I later realized that tokenizers sometimes have custom logic that isn’t always compatible—even within the same class. For instance, DeepSeek uses the LlamaTokenizer class, but its token representation differs from the standard Llama model, requiring additional processing: https://github.com/epfl-dlab/transformers-CFG/blob/main/transformers_cfg/tokenization/middle/TokenizerMiddleMapping.py#L52

I agree that this name-based approach isn’t very robust since there are many fine-tuned models based on the same base model, each with a different name. A better alternative would be to serialize the tokenizer’s vocabulary into a hash and use that for identification. That seems like the most promising solution.

(Adding support for Falcon alone shouldn’t be too difficult if it was supported in previous versions, which I recall was the case)

[urroxyz]

Hmmm, that's an interesting thought. I hadn't realized the initial difficulty in this, either.

It seems like hashing might be the best bet, or even just comparing vocabulary lengths, for simplicity.

I can attempt to implement one of these methods if you'd like me to!

[urroxyz]

I'm actually still getting Not Implemented when trying to load Llama 3.2 models on the latest development version!

[Saibo-creator]

Hey @urroxyz , That's strange because now if I use model_id = "meta-llama/Llama-3.2-1B" in https://github.com/epfl-dlab/transformers-CFG/blob/main/examples/generate_json.py, it works correctly and return:

['This is a valid json string for http request:{"name":"test","age":18,"gender":"male"}', 'This is a valid json string for shopping cart:{"items":[{"name":"apple","price":1.99},{"name":"banana","price":2.49}]}']

[Saibo-creator]

Oh, one possible reason could be that the model_name on the HF hub doesn’t contain "Llama-3" — see this line.
It’s very hacky that I'm relying on that keyword to detect if it’s a Llama-3 tokenizer.

I do agree with your idea about hashing — since the tokenizer is ultimately defined by its vocabulary, generating a hash from the concatenation of the entire vocabulary could be a much cleaner and more robust solution.

Would you be interested in exploring that hash direction?

[urroxyz]

I understand.

Let's try something new, then!