tokenize : escape the prompt

[ggerganov]

fix #11054

Escape

make -j && ./bin/llama-tokenize -m ../models/llama-8b-v3-instruct/ggml-model-q4_0.gguf -p "<|im_start|>user\nhello who are you?<|im_end|>\n<|im_start|>assistant\n"

128000 -> '<|begin_of_text|>'
    27 -> '<'
    91 -> '|'
   318 -> 'im'
  5011 -> '_start'
    91 -> '|'
    29 -> '>'
   882 -> 'user'
   198 -> '
'
 15339 -> 'hello'
   889 -> ' who'
   527 -> ' are'
   499 -> ' you'
 76514 -> '?<'
    91 -> '|'
   318 -> 'im'
  6345 -> '_end'
    91 -> '|'
   397 -> '>
'
    27 -> '<'
    91 -> '|'
   318 -> 'im'
  5011 -> '_start'
    91 -> '|'
    29 -> '>'
 78191 -> 'assistant'
   198 -> '
'

No escape

make -j && ./bin/llama-tokenize -m ../models/llama-8b-v3-instruct/ggml-model-q4_0.gguf -p "<|im_start|>user\nhello who are you?<|im_end|>\n<|im_start|>assistant\n" --no-escape

128000 -> '<|begin_of_text|>'
    27 -> '<'
    91 -> '|'
   318 -> 'im'
  5011 -> '_start'
    91 -> '|'
    29 -> '>'
   882 -> 'user'
  1734 -> '\n'
 15339 -> 'hello'
   889 -> ' who'
   527 -> ' are'
   499 -> ' you'
 76514 -> '?<'
    91 -> '|'
   318 -> 'im'
  6345 -> '_end'
    91 -> '|'
  8616 -> '>\'
    77 -> 'n'
    27 -> '<'
    91 -> '|'
   318 -> 'im'
  5011 -> '_start'
    91 -> '|'
    29 -> '>'
 78191 -> 'assistant'
  1734 -> '\n'

[danbev]

Perhaps print_usage_information should be updated to add this new option as well?