Add user-provided tokenizer/detokenizer functionality

[ilan-theodoro]

This addresses the issue #1765.

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[ngxson]

I think many things have changed since the issue was created, so the approach in this PR is not valid anymore.

User can now choose between using llama_tokenize or using their own implementation, so adding llama_set_custom_tokenizer provides no additional value.

Furthermore, adding custom tokenizer can potentially prevent the tokenizer API from being thread-safe.