What is the best way to count the tokens used by PlanAndExecuteAgentExecutor

[scottsuhy]

When you set "verbose: true" you can see all the tokens adding up. What is the best way to count all those tokens used?

const model = new ChatOpenAI({
  temperature: 0,
  modelName: "gpt-3.5-turbo",
  **verbose: true,**
});

const executor = PlanAndExecuteAgentExecutor.fromLLMAndTools({
  llm: model,
  tools,
});

[dosubot[bot]]

🤖

Hi @scottsuhy, good to see you again!

To count the tokens used by PlanAndExecuteAgentExecutor when verbose: true is set in the ChatOpenAI model, you can use the update_token_usage function in the openai.py file. This function updates the token usage by intersecting the keys from the response and the keys provided, and then adding the token usage from the response to the token usage dictionary.

Here is the function:

def update_token_usage(
    keys: Set[str], response: Dict[str, Any], token_usage: Dict[str, Any]
) -> None:
    """Update token usage."""
    _keys_to_use = keys.intersection(response["usage"])
    for _key in _keys_to_use:
        if _key not in token_usage:
            token_usage[_key] = response["usage"][_key]
        else:
            token_usage[_key] += response["usage"][_key]

This function is used in the _generate method of the OpenAI class:

def _generate(
        self,
        prompts: List[str],
        stop: Optional[List[str]] = None,
        run_manager: Optional[CallbackManagerForLLMRun] = None,
        **kwargs: Any,
    ) -> LLMResult:
        ...
        token_usage: Dict[str, int] = {}
        _keys = {"completion_tokens", "prompt_tokens", "total_tokens"}
        for _prompts in sub_prompts:
            ...
            else:
                response = completion_with_retry(
                    self, prompt=_prompts, run_manager=run_manager, **params
                )
                choices.extend(response["choices"])
                update_token_usage(_keys, response, token_usage)
        return self.create_llm_result(choices, prompts, token_usage)

In this method, the update_token_usage function is called with the keys {"completion_tokens", "prompt_tokens", "total_tokens"}, the response from the completion_with_retry function, and the token_usage dictionary. The token_usage dictionary is then returned as part of the LLMResult object.

The OpenAICallbackHandler class in the openai_info.py file also tracks token usage. The on_llm_end method of this class is called at the end of each language model run, and it updates the total tokens, prompt tokens, and completion tokens used by the model. It also calculates the cost of the tokens used based on the model used and the number of tokens. This class can be used to track token usage and cost in a more detailed manner.

You can find these files in the LangChain repository:

• libs/langchain/langchain/llms/openai.py
• libs/langchain/langchain/chat_models/openai.py
• libs/langchain/langchain/callbacks/openai_info.py

I hope this helps! Let me know if you have any other questions.

Sources

• libs/langchain/langchain/llms/openai.py
• libs/langchain/langchain/chat_models/openai.py
• libs/langchain/langchain/callbacks/openai_info.py
• libs/langchain/langchain/agents/agent_toolkits/openapi/planner.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[scottsuhy]

It's JavaScript... here is a cheap and easy way that seemed to work during debugging:

//save the original console.log function in originalLog.
//define a new console.log function that captures token count data from each log entry.

let originalLog = console.log;
let tokenCount = 0;

console.log = function(...args) {
    originalLog.apply(console, args);
    let logString = args.join(' ');
    let match = logString.match(/"totalTokens":\s(\d+)/);
    if (match) {
        tokenCount += parseInt(match[1], 10);
    }
};

const model = new ChatOpenAI({
  temperature: 0,
  modelName: "gpt-3.5-turbo",
  verbose: true,
});

const executor = PlanAndExecuteAgentExecutor.fromLLMAndTools({
  llm: model,
  tools,
});

console.log('Total tokens used:', tokenCount);
console.log = originalLog;

[devstein]

@scottsuhy Dosu's speaking Python because you're in the LangChain Python repository. I'd recommend closing this and opening a question in langchain-ai/langchainjs/discussions

[qaiwiz]

Is there any solution for token counting (completion, input,audio, ...) in agent execution?