vocab : BailingMoE : change possessive quantifiers to greedy

[CISC]

The possessive quantifiers are causing weird issues, and atomic grouping does not seem to be supported, so revert to greedy.

See following reports:

• llama : support BailingMoE (Ling) #12634 (comment)
• llama : support BailingMoE (Ling) #12634 (comment)
• https://huggingface.co/mradermacher/Ling-lite-GGUF/discussions/1

@bartowski1182 @nicoboss

[ggerganov]

Did you do tokenizer tests to make sure the results match with the reference tokenizer?

[CISC]

Did you do tokenizer tests to make sure the results match with the reference tokenizer?

I did some basic tests, but I will admit that I'm not entirely sure how to set up a proper test, any pointers?

[ggerganov]

You need to run:

python convert_hf_to_gguf_update.py <hf_token>

This will download reference tokenizers for all models to models/tokenizers and will generate test files in models/ggml-vocab-...inp/out.

After that, create a "vocab-only" GGUF model:

# this is for llama - update to create one for the Bailing model
python3 convert_hf_to_gguf.py models/tokenizers/llama-spm/ --outfile models/ggml-vocab-llama-spm.gguf --vocab-only

Run the test-tokenizer-0 tool using the GGUF vocab and generated test files.

[CISC]

Run the test-tokenizer-0 tool using the GGUF vocab and generated test files.

Ah, I missed the relationship between these files, I see now, added for future tests.

main : tokenizing: 'ggml-vocab-ling-plus.gguf.inp'
main : text size: 1944
main : tokenized in 1.466 ms (cpp)
main : tokens: 814
main : tokens written to 'ggml-vocab-ling-plus.gguf.inp.tokcpp'

Tests passed

[CISC]

@ggerganov gentle ping :)

[ggerganov]

Nice, but I didn't mean to commit the generated test files. At some point we stopped source controlling them because they add non negligible data (in this case 5MB). The idea is to just generate and test them locally.

[ggerganov]

Merge after removing the vocab files.

[bartowski1182]

huh, seems to be CUDA related? If I switch to CPU only it's able to tokenize no problem..

False alarms all around, was accidentally using an older build on my GPU... ignore me :) thanks so much for the fix!