QA with huge CSV type numeric Data gives Token Limit Error

[johnsudhir]

I am implementing one QA agent which can answer from the numeric data which is mostly CSV. But I have huge files which leads to the token limit error. There is numeric data so calculations need to be done on many files but we know the result gonna be token limit error.

Previously when I worked on a PDF-based QA bot where we feed mostly text-based documents I used Vector Database where based on similarity search I used to pass context and it was working fine.
But in numeric data we cannot do that as vector database cannot find that similarity.
How can we proceed with this? This really gonna help us to add new feature in langchain too.