Failed to calculate number of tokens when using streaming

[Luisotee]

Hello, I am having a problem where when I activate the streaming function it starts giving me the error: Failed to calculate number of tokens, falling back to approximate count Error: Unknown model. My code is:

Initializing the LLM:

import { LLMChain } from "langchain/chains";
import { ChatOpenAI } from "langchain/chat_models/openai";
import { ConversationSummaryMemory } from "langchain/memory";
import { PromptTemplate } from "langchain/prompts";
import {
  LLM_MODEL,
  OPENROUTER_API_KEY,
  OPEN_ROUTER_SYSTEM_MESSAGE,
} from "../constants";

const OPENROUTER_BASE_URL = "https://openrouter.ai";

const openRouterChat = new ChatOpenAI(
  {
    modelName: LLM_MODEL,
    streaming: true,
    temperature: 1,
    openAIApiKey: OPENROUTER_API_KEY,
  },
  {
    basePath: `${OPENROUTER_BASE_URL}/api/v1`,
  }
);

const memory = new ConversationSummaryMemory({
  memoryKey: "chat_history",
  inputKey: "input",
  llm: openRouterChat,
});
//${OPEN_ROUTER_SYSTEM_MESSAGE}
const systemMessageOpenRouter = PromptTemplate.fromTemplate(` 


## Current conversation:
    {chat_history}
    Human: {input}
    AI:

    `);

export const chain = new LLMChain({
  llm: openRouterChat,
  prompt: systemMessageOpenRouter,
  memory,
});

I am calling it in:

import { Message } from "whatsapp-web.js";
import { chain } from "../../clients/open-router";

export async function getCompletionWithOpenRouter(
  message: Message,
  context: string,
  streamingReply: Message
) {
  let messageWithContext = message.body;
  let tokenBuffer: string[] = [];

  let response = await chain.call(
    { input: messageWithContext },
    {
      callbacks: [
        {
          async handleLLMNewToken(token: string) {
            // Buffer the token
            tokenBuffer.push(token);

            // Update streamingReply with buffered tokens
            const updatedMessage = tokenBuffer.join("");

            // Edit the streamingReply with the updated message
            await streamingReply.edit(updatedMessage);
          },
        },
      ],
    }
  );

  return response.text;
}

[dosubot[bot]]

🤖

Hello @Luisotee! I'm Dosu, a friendly bot designed to lend a hand with bugs, answer questions, and assist you with contributing to the langchainjs project. While we're waiting for a human maintainer, I'm here to help out!

Based on the error message you're receiving, it seems like the model name provided is not recognized by the encodingForModel function, which is used to calculate the number of tokens in a given string for a specific language model. This function is called within the calculateMaxTokens function in the count_tokens.ts file. If the model name is not recognized, an error is thrown and caught in the catch block, and the error message is then logged to the console.

Here is the relevant code:

export const calculateMaxTokens = async ({
  prompt,
  modelName,
}: CalculateMaxTokenProps) => {
  let numTokens;

  try {
    numTokens = (
      await encodingForModel(getModelNameForTiktoken(modelName))
    ).encode(prompt).length;
  } catch (error) {
    console.warn(
      "Failed to calculate number of tokens, falling back to approximate count"
    );

    // fallback to approximate calculation if tiktoken is not available
    // each token is ~4 characters: https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them#
    numTokens = Math.ceil(prompt.length / 4);
  }

  const maxTokens = getModelContextSize(modelName);
  return maxTokens - numTokens;
};

To resolve this issue, ensure that the model name you are providing is recognized by the encodingForModel function. The function getModelNameForTiktoken is used to map the model name to a TiktokenModel type, which is then passed to encodingForModel. If the model name does not match any of the cases in getModelNameForTiktoken, it is returned as is. This could be the source of the problem if the model name is not recognized by encodingForModel.

Ensure that the model name you are using is one of the following: "gpt-3.5-turbo-16k", "gpt-3.5-turbo", "gpt-4-32k", "gpt-4", or any other model name that is recognized by the encodingForModel function.

In your case, the LLM_MODEL constant is set to "gpt-3.5-turbo-instruct", which should be a valid model name. If you're still encountering the issue, it might be due to a version mismatch or a bug in the library.

As for the streaming issue, it seems to be related to how streaming is handled in the LangChain framework. In the _generate method of the OpenAI class, when params.stream is true, a new Promise is created that waits for all the data to be received from the OpenAI API before it resolves. This is why handleLLMNewToken is receiving all tokens at once when the response is finished.

To handle tokens as they are generated, you would need to modify this code to resolve the Promise or handle the data as soon as each token is received, rather than waiting for the [DONE] message. The _streamResponseChunks method in the ChatOpenAI class is responsible for handling the streaming of responses. It uses the startStream method to initiate the stream and listens for the [DONE] message to close the stream. The handleLLMNewToken function is called within the _streamResponseChunks method for each chunk of data received from the stream. If you want to handle tokens as they are generated, you would need to modify the _streamResponseChunks method to yield each token as it is received, rather than waiting for the [DONE] message.

Also, it was mentioned in a previous issue that the .stream() method will not currently work properly for ConversationChain and LLMChain subclasses in general. You'll need to use the old callback approach. They want to deprecate it in favor of expression language but are still figuring out the right abstraction for memory.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• Failed to calculate number of tokens, falling back to approximate count
• Failed to calculate number of tokens with tiktoken, falling back to approximate count
• Streaming broken for ChatOpenAI in version 0.0.132 (and before)
• langchain-core/src/language_models/base.ts
• langchain/src/llms/openai.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Luisotee]

Streaming is already working, the program is running fine outside of this error message. The model that I am using is LLM_MODEL="google/palm-2-chat-bison-32k" but it also gives the same error with openai/gpt-3.5-turbo and others. I have not implementended any maximum token nor did any method to calculate the tokens that my LLM is using in chain.call.

[adesokanayo]

@Luisotee , were you able to get this fixed. I am having the same problem with my model on azure open Ai

[Luisotee]

Yes, some update fixed it somewhen

[imtiger]

@Luisotee , were you able to get this fixed. I am having the same problem with my model on azure open Ai

I also have the same error with below code:
const chat = new ChatOpenAI(
{
modelName: "openai/gpt-3.5-turbo",
// modelName: "tiiuae/falcon-40b-instruct",
// modelName: "anthropic/claude-instant-v1",
// modelName: "anthropic/claude-2",
// modelName: "google/palm-2-chat-bison",
temperature: 0.8,
maxTokens: 300,
streaming: true,
openAIApiKey: OPENROUTER_API_KEY
},

but if I change the modename from openai/gpt-3.5-turbo to gpt-3.5-turbo,it works, you can try ,just delete the prefix of modename

[Luisotee]

Have you changed the basePath? Mine is like this:

 const openRouterChat = new ChatOpenAI(
    {
      modelName: llmModel,
      streaming: true,
      temperature: 0.7,
      openAIApiKey: OPENROUTER_API_KEY,
    },
    {
      basePath: `${OPENROUTER_BASE_URL}/api/v1`,
    }
  );

Also make sure you are using the latest version of langchain, as that was what was causing the problem for me.

If you want you can checkout how I made it work here code related to langchain is in src/clients/open-router.ts and src/handlers/llm-models/completion-open-router.ts