How to optimize the cost of each Ask?

[joaolovatti]

Context / Scenario

• I have noticed that each request I make to the \ask endpoint is generating a cost of 2 cents using the OpenAI API.

• It has been using a large amount of tokens as context.

• Compared to smaller uses like asking on ChromaDB, I found that Kernel Memory is using a higher amount of tokens.

Question

• Do you have any tips on how to reduce the cost?

• Can I limit the amount of context tokens used for the LLM? Or limit the number of relevant partitions returned?

[amitchaudhary]

+1

[dluc]

One way to reduce the tokens used is using the minRelevance parameter. The default value is 0 so the RAG prompt will include as many chunks as possible.

By increasing that value, you can exclude chunks with lower relevance.
There's no magic value, and the exact value depends on your data, on your questions, and the storage engine.

If you're using only vector search with cosine similarity, not using hybrid search nor custom search algos, you could start with a value like 0.75 and see if that helps. Note that by increasing the value, the RAG prompt will get smaller and smaller, giving LLMs less and less ground information to answer your questions.