-
|
Hello, just a quick question what is the purpose of using the tokenization that is ported vs. Microsoft Tokenizer Library? Thank you 🙏 |
Beta Was this translation helpful? Give feedback.
Answered by
dluc
Dec 28, 2023
Replies: 1 comment
-
|
no particular reason, the library in the repo was developed earlier (used to be here), it's a direct port of OpenAI lib and has been optimized over the time for performance. Tokenizers can be injected though, so if one prefers using a different implementation that's possible. |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
dluc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
no particular reason, the library in the repo was developed earlier (used to be here), it's a direct port of OpenAI lib and has been optimized over the time for performance. Tokenizers can be injected though, so if one prefers using a different implementation that's possible.