Help: Chatbot steps out of role (using: ConversationSummaryBufferMemory())

[ckx2780]

Hi all,

I'm writing a script that takes as input an appliance manual (e.g. from a dishwasher) and a certain issue (e.g. "Dishwasher is not draining") and should output resolution steps based on the manual in a conversational manner. I am using few-shot learning in the prompt. This is how I generate the answer:

# Generate the initial answer
llm = ChatOpenAI(
    temperature=0,
    model_name= "gpt-4" 
)
memory = ConversationSummaryBufferMemory(
    llm=llm,
    output_key='answer',
    memory_key='chat_history',
    return_messages=True
)
pdf_qa = ConversationalRetrievalChain.from_llm(
    llm=llm,
    memory=memory,
    chain_type="stuff",
    retriever=vectordb.as_retriever(), # embeddings created using Chroma.from_documents()
    return_source_documents=True,
    get_chat_history=lambda h : h,
    verbose=False
)

result = pdf_qa({"question": prompt})
result["answer"]

The answer is relevant:
"This could be due to a clog or blockage. Here's what you can do:\n\n- Check the bottom of the dishwasher for any food particles or small objects that might be blocking the drain.\n\n- Inspect the drain hose under the sink. It should be high looped. If it's not, water from the sink could be going into the dishwasher.\n\n- Check the filter at the bottom of the dishwasher. If it's clogged, clean it out.\n\nIf the issue persists, I will go ahead and schedule a technician for you. Try these solutions and tell me... did it work?"

Then, when generating the follow-up answer...

# Generate follow-up answers
prompt = "It did not work" 
result = pdf_qa({"question": prompt})
result["answer"]

... the chatbot suddenly seems to be lost:
"As an AI, I don't have personal experiences. However, I can help you troubleshoot further if the proposed solutions for your dishwasher issue didn't work. Could you please specify what the issue is?"

Is this an issue because of the memory object I'm using? Or is it a matter of prompt engineering? Would appreciate any help!

[dosubot[bot]]

Answer generated by a 🤖

Answer

From your description, it seems like you're having an issue with the chatbot losing context during the conversation. You're using the ConversationSummaryBufferMemory object, and while generating a follow-up answer, the chatbot seems to forget the previous context and steps out of its role.

The proposed solution to this issue is to switch to using the ConversationBufferMemory object instead of ConversationSummaryBufferMemory. The ConversationBufferMemory is designed to maintain a buffer of the conversation, which could be more effective in preserving the context of the conversation in your case.

Here's how you can modify your code:

from langchain.prompts import (
    ChatPromptTemplate,
    MessagesPlaceholder,
    SystemMessagePromptTemplate,
    HumanMessagePromptTemplate
)
from langchain.chains import ConversationChain
from langchain.chat_models import ChatOpenAI
from langchain.memory import ConversationBufferMemory, ReadOnlySharedMemory

prompt = ChatPromptTemplate.from_messages([
    SystemMessagePromptTemplate.from_template(
        "The following is a friendly conversation between a human and an AI. The AI is talkative and "
        "provides lots of specific details from its context. If the AI does not know the answer to a "
        "question, it truthfully says it does not know."
    ),
    MessagesPlaceholder(variable_name="history"),
    HumanMessagePromptTemplate.from_template("{input}")
])

llm = ChatOpenAI(temperature=0)
memory = ConversationBufferMemory(return_messages=True)
readonlymemory = ReadOnlySharedMemory(memory=memory)
pdf_qa = ConversationChain(memory=readonlymemory, prompt=prompt, llm=llm)

After this modification, you can generate the initial and follow-up answers as you did before. This change should help the AI maintain the context of the conversation and provide more relevant responses.

Please note that this is one potential solution based on the information provided. If this doesn't solve your issue, there might be other factors at play that we would need to investigate further.

If you find this solution helpful and believe it could benefit other users, I encourage you to make a pull request to update the documentation. This way, others facing a similar issue can also benefit from your experience.

Sources

• docs/extras/modules/memory/how_to/conversational_customization.ipynb
• docs/extras/modules/agents/how_to/chatgpt_clone.ipynb
• docs/extras/modules/memory/how_to/agent_with_memory.ipynb
• docs/snippets/modules/memory/how_to/summary.mdx
• docs/snippets/get_started/quickstart/memory_chat_models.mdx
• docs/extras/modules/agents/how_to/sharedmemory_for_tools.ipynb

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

[ckx2780]

Thanks for the suggestions. Unfortunately, it does not work. When running the follow-up question, I get:

"I'm sorry to hear that the previous steps didn't resolve your issue. Let's try a few more things. Could you please tell me what appliance you're having trouble with?"
Even though the appliance (dishwasher) is clearly mentioned in the original question.

In addition, ConversationChain doesn't allow me to pass a retriever (with the PDFs), like ConversationalRetrievalChain.from_llm does.

Do you have any other suggestions I can try?

[dosubot[bot]]

Answer generated by a 🤖

Answer

I understand that you're developing a script using the LangChain Python framework to create a conversational chatbot. You're using the ConversationalRetrievalChain.from_llm method for generating initial and follow-up answers. However, you're encountering an issue where the chatbot loses context during the conversation. Despite trying ConversationBufferMemory and ConversationChain, the problem persists.

To address this, one potential solution could be to manually manage the conversation history. This would involve appending each new question and the corresponding generated answer to the conversation history after each interaction. This way, the entire conversation history will be available for the next prompt, which should help maintain the context of the conversation.

Here's how you can modify your code:

# Initialize the conversation history
conversation_history = []

# Generate the initial answer
llm = ChatOpenAI(
    temperature=0,
    model_name= "gpt-4" 
)
memory = ConversationSummaryBufferMemory(
    llm=llm,
    output_key='answer',
    memory_key='chat_history',
    return_messages=True
)
pdf_qa = ConversationalRetrievalChain.from_llm(
    llm=llm,
    memory=memory,
    chain_type="stuff",
    retriever=vectordb.as_retriever(), # embeddings created using Chroma.from_documents()
    return_source_documents=True,
    get_chat_history=lambda h : h,
    verbose=False
)
result = pdf_qa({"question": prompt})
result["answer"]

# Append the initial question and answer to the conversation history
conversation_history.append({"question": prompt, "answer": result["answer"]})

# Generate follow-up answers
prompt = "It did not work" 
result = pdf_qa({"question": prompt, "chat_history": conversation_history})
result["answer"]

# Append the follow-up question and answer to the conversation history
conversation_history.append({"question": prompt, "answer": result["answer"]})

This approach should ensure that the entire conversation history is passed as context for each new prompt, which should help in maintaining the context of the conversation.

Please note that this is a workaround and might not be the most efficient solution. It would be ideal to investigate why the ConversationBufferMemory class is not working as expected in your case.

If you find this solution helpful and believe it could benefit other users, I encourage you to make a pull request to update the LangChain documentation. This way, other developers facing similar issues can also benefit from your experience.

Sources

• langchain/chains/conversation/base.py

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.