Add user-provided tokenizer/detokenizer functionality#10131

ilan-theodoro wants to merge 1 commit intoggml-org:masterggml-org/llama.cpp:masterfrom

ilan-theodoro:masterilan-theodoro/llama.cpp:masterCopy head branch name to clipboard