How to get token information(response_metadata) from the result in stream

[JANGSOONMYUN]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

# Define your response schemas
response_schemas = [
    ResponseSchema(name="name", description="A good name for the company"),
    ResponseSchema(name="industry", description="The industry the company is in")
]

# Create the parser from the response schemas
output_parser = StructuredOutputParser.from_response_schemas(response_schemas) 

# Define the prompt template
prompt_template_1 = ChatPromptTemplate(
    messages=[
        HumanMessagePromptTemplate.from_template(
            "What is a good name for a company that makes {product}? "
            "Also, specify the industry.\n{format_instructions}"
        )
    ],
    input_variables=["product"],
    partial_variables={"format_instructions": output_parser.get_format_instructions()}
)

# Create the chain
chain_1 = (
    prompt_template_1
    | llm
)
chain_2 = (
    prompt_template_1
    | llm
)

map_chain = RunnableParallel({"chain_1":chain_1, "chain_2":chain_2})

# Execute the chain and get the result
product = "eco-friendly water bottles"

from langchain_core.runnables.utils import AddableDict
result_1 = map_chain.stream({"product": product})

tot_str = {'chain_1': '', 'chain_2': ''}
for s in result_1:
    if 'chain_1' in s:
        tot_str['chain_1'] += s['chain_1'].content
    if 'chain_2' in s:
        tot_str['chain_2'] += s['chain_2'].content

# how to get token information
print(result_1.response_metadata)

Description

I would like to know how to get correct token information after getting the result via streaming.

If the result without streaming, it includes response_metadata but if with streaming, it doesn't have response_metadata.

I want to check the currently used token size when I use stream option.

System Info

langchain==0.2.9
langchain-community==0.2.9
langchain-core==0.2.22
langchain-openai==0.1.10
langchain-text-splitters==0.2.2
langchainhub==0.1.21

linux(ubuntu)
python 3.9

[dosubot[bot]]

To get token information (response_metadata) from the result when using the stream option in LangChain, you can access the response_metadata attribute of the AIMessageChunk object yielded by the stream method. Here is an example of how you can do this:

import sys

# Assuming chat is an instance of ChatBaichuan
messages = [
    ("system", "You are a helpful assistant."),
    ("human", "Tell me a joke."),
]

for chunk in chat.stream(messages):
    sys.stdout.write(chunk.message.content)
    sys.stdout.flush()
    print(chunk.message.response_metadata)

In this example, chunk.message.response_metadata will contain the token information (response_metadata) for each chunk in the stream ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

To get response_metadata when using the map_chain.stream method, you need to access the response_metadata attribute of each chunk in the stream. Here is how you can modify your code to extract and print the response_metadata:

result_1 = map_chain.stream({"product": product})

tot_str = {'chain_1': '', 'chain_2': ''}
response_metadata = {'chain_1': [], 'chain_2': []}

for s in result_1:
    if 'chain_1' in s:
        tot_str['chain_1'] += s['chain_1'].content
        response_metadata['chain_1'].append(s['chain_1'].response_metadata)
    if 'chain_2' in s:
        tot_str['chain_2'] += s['chain_2'].content
        response_metadata['chain_2'].append(s['chain_2'].response_metadata)

print("Chain 1 Response Metadata:", response_metadata['chain_1'])
print("Chain 2 Response Metadata:", response_metadata['chain_2'])

In this code, response_metadata is collected for each chunk and stored in a list for each chain. This allows you to access and print the metadata after the streaming is complete ^[1][2].

To continue talking to Dosu, mention @dosu.

[JANGSOONMYUN]

@dosu

The result is
{'finish_reason': 'stop', 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'fp_f33667828e'}
{'finish_reason': 'stop', 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'fp_f33667828e'}

how to know the prompt token size for input and output?

[dosubot[bot]]

To determine the prompt token size for both input and output from the result, you need to ensure that the response_metadata includes detailed token usage information. The response_metadata should contain keys like prompt_tokens, completion_tokens, and total_tokens.

Here is how you can modify your code to extract and print the token usage information:

result_1 = map_chain.stream({"product": product})

tot_str = {'chain_1': '', 'chain_2': ''}
response_metadata = {'chain_1': [], 'chain_2': []}

for s in result_1:
    if 'chain_1' in s:
        tot_str['chain_1'] += s['chain_1'].content
        response_metadata['chain_1'].append(s['chain_1'].response_metadata)
    if 'chain_2' in s:
        tot_str['chain_2'] += s['chain_2'].content
        response_metadata['chain_2'].append(s['chain_2'].response_metadata)

# Print the response metadata for each chain
print("Chain 1 Response Metadata:", response_metadata['chain_1'])
print("Chain 2 Response Metadata:", response_metadata['chain_2'])

# Extract and print token usage information
for chain, metadata_list in response_metadata.items():
    for metadata in metadata_list:
        if 'token_usage' in metadata:
            token_usage = metadata['token_usage']
            prompt_tokens = token_usage.get('prompt_tokens', 0)
            completion_tokens = token_usage.get('completion_tokens', 0)
            total_tokens = token_usage.get('total_tokens', 0)
            print(f"{chain} - Prompt Tokens: {prompt_tokens}, Completion Tokens: {completion_tokens}, Total Tokens: {total_tokens}")

In this code, response_metadata is collected for each chunk and stored in a list for each chain. After the streaming is complete, the code extracts and prints the token usage information from the response_metadata if it is available.

If the response_metadata does not include token usage information, you may need to ensure that the model and API you are using support and return this information. For example, OpenAI's API typically includes token usage in the response metadata, but it may vary depending on the specific model and configuration ^[1][2].

To continue talking to Dosu, mention @dosu.

[JANGSOONMYUN]

@dosu
There is no token_usage so I cannot check the token size
In metadata_list, there are [finish_reason, model_name, system_fingerprint] only

[dosubot[bot]]

To check the prompt token size for input and output when the response_metadata only contains finish_reason, model_name, and system_fingerprint, you can use the tiktoken package to calculate the number of tokens. Here is an example of how you can do this:

import tiktoken

def get_num_tokens(string: str, model_name: str) -> int:
    encoding = tiktoken.encoding_for_model(model_name)
    num_tokens = len(encoding.encode(string))
    return num_tokens

# Example usage
input_text = "Your input text here"
output_text = "Your output text here"
model_name = "gpt-3.5-turbo"  # Replace with your model name

prompt_tokens = get_num_tokens(input_text, model_name)
completion_tokens = get_num_tokens(output_text, model_name)

print(f"Prompt Tokens: {prompt_tokens}")
print(f"Completion Tokens: {completion_tokens}")

If you are using LangChain, you can track token usage through the AIMessage.usage_metadata attribute. This attribute, when populated, will be a dictionary with standard keys such as "input_tokens" and "output_tokens". Here is an example using LangChain:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-3.5-turbo-0125")
openai_response = llm.invoke("hello")
usage_metadata = openai_response.usage_metadata

print(f"Input Tokens: {usage_metadata['input_tokens']}")
print(f"Output Tokens: {usage_metadata['output_tokens']}")

For streaming, you can enable token usage tracking by setting stream_usage=True when instantiating the chat model:

llm = ChatOpenAI(model="gpt-3.5-turbo-0125", stream_usage=True)

aggregate = None
for chunk in llm.stream("hello", stream_usage=True):
    print(chunk)
    aggregate = chunk if aggregate is None else aggregate + chunk

print(aggregate.content)
print(aggregate.usage_metadata)

This code will output the number of tokens used for the prompt (input) and the completion (output) ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.