How to support complex queries like doc count

[syonip]

There's a certain class of questions that seems to fail/generate nonsense. Examples from user chat:

• How many documents are in the database?
• How many documents have been signed after March 2025?
• Give me a list of all contracts between company X and company Y

These seem to be more in the area of generating a proper search query. I don't even know if this falls under RAG. But the user requires supporting this.

Any idea or resource would be appreciated.
Thanks.

[pamelafox]

I agree that the default approach would not work well for those queries. Generally, if you knew the common types of queries, you could use the query rewriting step to reroute user requests to handle them. I blogged about that here:
https://blog.pamelafox.org/2024/03/rag-techniques-using-function-calling.html

It also sounds like your questions might be better answered by a relational database, so you might consider bringing in a SQL database to answer some questions. Just be careful with arbitrary text to SQL - safer when you can constraint the possible filters (WHERE clauses).

[syonip]

Thanks, I read your blog post.
It seems like the solution is:

1. Create another index or SQL table to hold various metadata fields,
2. Use function calling to decide which question requires free text query and which requires an SQL query.

Is that the general direction?
Is agentic retrieval relevant here?

Thanks.

[jack-vinitsky]

An idea that I have been thinking is to generate a "document" that contains this aggregate information about the entire corpus/index used for the chatbot and then add it to index. Obviously you would need to recreate this document when you have a change to your corpus but for some use cases it may be worthwhile.