vocab: refactor tokenizer to reduce the overhead of creating multi times tokenizer#9449

ggerganov merged 12 commits intoggml-org:masterggml-org/llama.cpp:masterfrom

kylo5aby:refactor-tokenizerkylo5aby/llama.cpp:refactor-tokenizerCopy head branch name to clipboard