Shouldn't we use more than one different document? when using ConversationalRetrievalChain.

[SAIVENKATARAJU]

Hi
I have stored to different documents in vectore store. one is alphabet-revenue.pdf and another different pdf. To get the conversational experience. I am using ConversationalRetrievalChain. I have asked the following questions.

• What were alphabet revenues in 2022
• What about costs and expenses?
• Is this more than in 2021?
  So, for the last two questions it actually giving answers form second pdf(I checked the source). it seems, LLM is not considering as follow-up questions.

My code

llm = ChatVertexAI(temperature=1, max_output_tokens=1500)
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
conversational_retrieval = ConversationalRetrievalChain.from_llm(
        llm=llm, retriever=faiss_index.as_retriever(), memory=memory
    )
result = conversational_retrieval({"question": query})

I am thinking it's not answering from chat_history. is there any mistake I am doing here.

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out and providing a detailed explanation of your issue. It seems like you're experiencing a problem where the ConversationalRetrievalChain is not considering follow-up questions and retrieving answers from the second PDF instead of the chat history.

This issue might be due to the way the question_generator is implemented in the ConversationalRetrievalChain. The question_generator is used to generate a new standalone question for retrieval. If the chat history is not empty, the question_generator is run with the current question and chat history. If the chat history is empty, the original question is used. Here is the relevant code snippet:

get_chat_history = self.get_chat_history or _get_chat_history
chat_history_str = get_chat_history(inputs["chat_history"])
if chat_history_str:
    callbacks = _run_manager.get_child()
    new_question = self.question_generator.run(
        question=question, chat_history=chat_history_str, callbacks=callbacks
    )
else:
    new_question = question

If the question_generator is not properly considering the chat history when generating the new question, it could lead to the issue you're experiencing. To fix this issue, you would need to modify the question_generator to properly consider the chat history when generating the new question.

Also, I found a similar issue in the LangChain repository: ConversationalRetrievalChain did not look into chat history while making an answer. The suggested solution is to update to the latest version of the software as it may have fixed this issue. The latest version of LangChain is v0.0.301.

I hope this helps! If you have any other questions or need further clarification, please don't hesitate to ask.

Sources

• ConversationalRetrievalChain did not look into chat histroy while making an answer
• libs/langchain/langchain/chains/conversational_retrieval/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.