Counting tokens after response in stream.

[Alerinos]

There is no display of used tokens in CreateCompletionAsStream, I would like to record how many tokens I used each time.
Also Usage is null.

[Alerinos]

I just see that there is such a thing

TokenizerGpt3.TokenCount()

If I understand correctly, you can count the entire prompt (including previous messages) at the end of the result, and then answer and return the finished result to the user. What do you think about something like this?

Example:

        var completionResult = openAiService.ChatCompletion.CreateCompletionAsStream(new ChatCompletionCreateRequest
        {
            Messages = messageSend,
            Temperature = (float)s.Temperature,
            Model = model_GPT,
        });

        await foreach (var completion in completionResult)
        {
            if (completion.Successful)
            {
                ...
            }
        }
        
        var countPrompt = completionResult.Tokens.Prompt();
        var countOutput = completionResult.Tokens.Completion();

[Alerinos]

590 + 92 = 682

Code:

  var count = messageSend.Select(x => TokenizerGpt3.TokenCount(x.Content)).ToList().Sum();
  var count2 = TokenizerGpt3.TokenCount(OutputAI.Message);
  Console.WriteLine($"{count} + {count2} = {count + count2}");

Biling:

Is something wrong?

https://github.com/betalgo/openai/blob/ebfb7af6f6d1db6c2ab335eda7054e026542f194/OpenAI.SDK/Tokenizer/GPT3/TokenizerGpt3.cs

According to the AI analysis, there are such possibilities:

1. The BytePairEncoding function could be creating new tokens that were not present in the original text, resulting in an overestimation of the token count. This is because BytePairEncoding merges frequently occurring character pairs into a single token, which could introduce new tokens that were not present in the original text. For example, if the text contained the word "apple", BytePairEncoding might merge the "p" and "l" to create a new token "pl", even though "pl" does not appear in the original text. This would result in an overestimation of the token count.

2. The regular expression used by the EncodingRegex variable might be splitting tokens in a way that is not accurate for the given data. For example, if the regular expression is splitting on whitespace characters, but the text contains non-standard whitespace characters that are not matched by the regex, then some tokens might be split into multiple tokens, resulting in an overestimation of the token count.

3. The BpeCache dictionary might be caching incorrect values for certain tokens, resulting in an overestimation of the token count. If the BytePairEncoding function creates the same token multiple times, but the BpeCache only stores the result for the first instance of the token, then subsequent instances of the token would be counted as new tokens, resulting in an overestimation of the token count.

[kayhantolga]

TokenizerGpt3.TokenCount does not use the OpenAI service, but it employs a similar calculation method. Tokens may differ depending on character encoding or environment. For example, if you use Windows, it uses /r/n for the "end of line" character, which will display some extra tokens.

The OpenAI stream endpoint does not provide usage information. I like the idea of adding these values by calculating them in the Response Model, but this may cause misunderstandings among developers and lead them to think that the origin of the data is the API.

I am going to convert this issue to discussion, so we can talk about it if can find an easier way to implement a tokenizer.

Again, the diff in your example is probably end-of-line characters, if it is not, please share the content so we can test it.

[Alerinos]

TokenizerGpt3.TokenCount does not use the OpenAI service, but it employs a similar calculation method. Tokens may differ depending on character encoding or environment. For example, if you use Windows, it uses /r/n for the "end of line" character, which will display some extra tokens.

It seems to me that we should be able to detect such a scenario and character encoding. Alternatively, we can send everything in UTF-8 which seems to me to be resistant to these problems and all cultures of the world.
https://learn.microsoft.com/en-us/dotnet/api/system.text.encoding.getencoding?view=net-7.0

If we make the same provider for sending messages as to counting tokens, they should match.

I am going to convert this issue to discussion, so we can talk about it if can find an easier way to implement a tokenizer.

Until we get the perfect solution, we can call the function completionResult.ExperimentalToken.Prompt();.

This will cause the user to look at the documentation or see a summary

Again, the diff in your example is probably end-of-line characters, if it is not, please share the content so we can test it.

I will do some tests and let you know how it turned out, if there is still a problem, I will prepare a demo repository

[dannyr-git]

static async Task<string> SubmitChat(List<ChatMessage> chat)
{
    var _openAiService = new OpenAIService(new OpenAiOptions()
    {
        ApiKey =  "sk-9DJsqWhpjRSO9muXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
    }); ;

    var completionResult = await _openAiService.ChatCompletion.CreateCompletion(new ChatCompletionCreateRequest
    {
        Messages = chat,
        Model = Models.ChatGpt3_5Turbo,
        MaxTokens = 300,
    });
    Console.WriteLine($"Completion Tokens={completionResult.Usage.CompletionTokens},Prompt Tokens={completionResult.Usage.PromptTokens}, Total Tokens={completionResult.Usage.TotalTokens}");
    return completionResult.Choices.First().Message.Content;
}


var promptStr = "I want to go to the supermarket and buy apples.";
int test1 = 0;
int test2 = 0;

Console.WriteLine(">>>TESTx1");
var newchat = new List<ChatMessage>();

for (int i = 0; i < 1; i++)
{
    newchat.Add(ChatMessage.FromUser(promptStr));
    test1 += TokenizerGpt3.TokenCount(promptStr);
    test2 += TokenizerGpt3.TokenCount(promptStr, true);
}
Console.WriteLine($"test1={test1}, test2={test2}");
await SubmitChat(newchat);

Console.WriteLine(">>>TESTx50");
newchat.Clear();
test1 = test2 = 0;

for (int i = 0; i < 50; i++)
{
    newchat.Add(ChatMessage.FromUser(promptStr));
    test1 += TokenizerGpt3.TokenCount(promptStr);
    test2 += TokenizerGpt3.TokenCount(promptStr, true);
}
Console.WriteLine($"test1={test1}, test2={test2}");

await SubmitChat(newchat);

Console.WriteLine(">>>TESTx25");
newchat.Clear();
test1 = test2 = 0;

for (int i = 0; i < 25; i++)
{
    newchat.Add(ChatMessage.FromUser(promptStr));
    test1 += TokenizerGpt3.TokenCount(promptStr);
    test2 += TokenizerGpt3.TokenCount(promptStr, true);
}
Console.WriteLine($"test1={test1}, test2={test2}");
await SubmitChat(newchat);

Results :

>>>TESTx1
test1=11, test2=11
Completion Tokens=184,Prompt Tokens=19, Total Tokens=203
>>>TESTx50
test1=550, test2=550
Completion Tokens=12,Prompt Tokens=803, Total Tokens=815
>>>TESTx25
test1=275, test2=275
Completion Tokens=14,Prompt Tokens=403, Total Tokens=417

Threw this test together b/c I couldn't figure out why my app had such large discrepancies as i keep running into max token limits.

TokenizerGpt3.TokenCount() probably shouldn't ever be used

[Frogley]

For now TokenizerGpt3 only support encoding p50k_base, while gpt-3.5-turbo is using encoding cl100k_base. So you are right, TokenizerGpt3.TokenCount() shouldn't be used for gpt-3.5-turbo model. see the details
By the way, you've exposed your API Key in the code, please change it ASAP.

[dannyr-git]

No I didn't, I replaced most of it with xxxx's :)

[dannyr-git]

Just found:
https://github.com/openai/tiktoken
https://github.com/aiqinxuancai/TiktokenSharp

https://github.com/dmitry-brazhenko/SharpToken looks promising too

[richardtallent-erm]

Hi,

I'm new to using this library, and I'm not sure I'm following the conversation.

I'll probably use Microsoft's tokenizer library to get a rough estimate of token use so I can batch my requests, but I'd like to keep a full accounting of the actual tokens used based on OpenAI's counts, since that's what they'll be billing against.

In the OpenAI API docs, they say the Usage object in the response should include the prompt and completion token counts:

https://platform.openai.com/docs/guides/gpt/chat-completions-api

But I'm using streamed completion, and in all results from await foreach(var completion in CreateCompletionAsStream(...) using this library, the Usage object is null.

Is OpenAI not sending this information when stream is set to true, even on the final partial completion?