Question and Answer Based from different data sources

[nacartwright]

I'm putting together a question and answer agent. I have two data sources:

1. One for an upcoming conference that lists all the sessions, schedules, etc.
2. Another one that is over our website so people can ask questions on products/services.

I'm using the Google Generative AI Enterprise Search for the initial search (https://github.com/nacartwright/google-gen-ai-search-retriever). I initially had the Generative AI search setup to index the two websites. For some reason, I noticed that if you ask the search engine a question it would prefer the wrong website. Currently, there are no ways of applying weights within GCP for the search engine.

So, I want to be able to split up the search indexes into two different retrievers. I tried to use MultiRetrievalQAChain, however, the description doesn't seem to be able catch everything that is possible. For example, I have a description for the even that is coming up but when I ask a general question that doesn't specify the event name it will not select the tool and will give a generic answer from the language model.

Is there a way that I can have two chains then compare the output from to see if it matches the user query before finishing the agent?

If it doesn't find a match, then I don't want it to answer. I've tried a few different methods but either cannot get it to search the correct search index or it gives a generic answer from the LLM model.

Also, can you restrict the agent to only answer questions based off of the retrievers?