fix tokenizer for JetBrain Mellum#15045

CISC merged 4 commits intoggml-org:masterggml-org/llama.cpp:masterfrom

csabakecskemeti:JB_mellum_tokenizer_supportcsabakecskemeti/llama.cpp:JB_mellum_tokenizer_supportCopy head branch name to clipboard