How to calculate the cost and tokens when using Faiss vector database with openai through LangChain

[AhmedElmaghraby207]

I have built a chatbot using OpenAI Langchain and Faiss vector database and I need to know how to calculate the used tokens in all process [OpenAIEmbeddings, prompt template, user message, assistant message] Pls advice me in details to allow us go with this chatbot to the next stage

[dosubot[bot]]

Answer generated by a 🤖

Answer

I understand that you've built a chatbot using OpenAI's LangChain and the Faiss vector database, and you're looking to calculate the number of tokens used in the process. This includes tokens used in OpenAIEmbeddings, the prompt template, user message, and assistant message.

To achieve this, you can utilize the get_openai_callback() context manager provided by LangChain. This context manager is designed to track the token usage for specific calls made within its scope.

Here's how you can use it:

from langchain.llms import OpenAI
from langchain.callbacks import get_openai_callback

llm = OpenAI(model_name="text-davinci-002", n=2, best_of=2)

with get_openai_callback() as cb:
    result = llm("Your query here")
    print(cb)

In this code, any calls made within the get_openai_callback() context manager will have their token usage tracked. If you want to track multiple calls in sequence, you can do so like this:

with get_openai_callback() as cb:
    result = llm("Your first query here")
    result2 = llm("Your second query here")
    print(cb.total_tokens)

If you're using a chain or agent with multiple steps, it will track all those steps. Here's an example:

from langchain.agents import load_tools
from langchain.agents import initialize_agent
from langchain.agents import AgentType
from langchain.llms import OpenAI

llm = OpenAI(temperature=0)
tools = load_tools(["serpapi", "llm-math"], llm=llm)
agent = initialize_agent(
    tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True
)

with get_openai_callback() as cb:
    response = agent.run(
        "Your complex query here"
    )
    print(f"Total Tokens: {cb.total_tokens}")
    print(f"Prompt Tokens: {cb.prompt_tokens}")
    print(f"Completion Tokens: {cb.completion_tokens}")
    print(f"Total Cost (USD): ${cb.total_cost}")

In this example, cb.total_tokens gives the total tokens used, cb.prompt_tokens gives the tokens used in the prompt, cb.completion_tokens gives the tokens used in the completion, and cb.total_cost gives the total cost in USD.

The get_openai_callback() context manager is also concurrency safe. This means you can initiate concurrent runs from within the context manager and it will correctly count the tokens used.

This solution should help you calculate the cost and tokens when using the Faiss vector database with OpenAI through LangChain. If you find this information helpful and believe it could benefit others, I encourage you to make a pull request to update the documentation. This way, other users facing similar issues can also benefit from your findings.

Sources

• docs/extras/modules/model_io/models/llms/token_usage_tracking.ipynb
• libs/langchain/langchain/callbacks/openai_info.py
• docs/extras/modules/callbacks/token_counting.ipynb

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

[Enliven26]

store = FAISS.from_texts(array_of_string, OpenAIEmbeddings(), metadatas=array_of_metadata)

is there a way to get the total token from the vectorstore?