Is Microsoft.SemanticKernel.Connectors.AI.OpenAI still in use?
#6481
-
|
I want to use tokenizer like GPT3Tokenizer to count number of tokens. But I don't see the library being maintained., last update was 6 months ago. Do we have any other library that I can use it do the same job done? |
Beta Was this translation helpful? Give feedback.
Answered by
stephentoub
Jun 2, 2024
Replies: 1 comment
-
|
The library was renamed to Microsoft.SemanticKernel.Connectors.OpenAI. However, the GPT3Tokenizer type was removed. Use the Microsoft.ML.Tokenizers library for tiktoken tokenization. |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
sophialagerkranspandey
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The library was renamed to Microsoft.SemanticKernel.Connectors.OpenAI.
However, the GPT3Tokenizer type was removed. Use the Microsoft.ML.Tokenizers library for tiktoken tokenization.