"Continue Generating" feature using ConversationalRetrievalChain

[7empestGit]

I'm building a chat bot that answers questions based on context and I have set LLM's token limit to 100. But what should I do if I want to get LLM's answer so that it's literally the continuation of the previous answer? First thing that came to my mind was just trying to type "continue" or "continue generating" as a prompt while using chat_history variable that includes previous question and LLM's previous answer, but as a result I've got the output that is not the one I wanted (it just regenerates the previous answer in other form)

[dosubot[bot]]

🤖

Hello,

Based on your description, it seems like you want the Language Learning Model (LLM) to generate a continuation of its previous answer, rather than regenerating the previous answer in a different form. This can be achieved by modifying the way the chat history is used in the question generation process.

In the BaseConversationalRetrievalChain class, the question_generator chain is used to generate a new question for retrieval based on the current question and chat history. This new question is then used to retrieve relevant documents and generate a final response.

If you want the LLM to generate a continuation of its previous answer, you need to modify the question_generator chain to include the previous answer in the generated question. This can be done by modifying the PromptTemplate used in the question_generator chain.

Here is an example of how you can modify the PromptTemplate:

# This controls how the standalone question is generated.
# Should take `chat_history`, `question`, and `previous_answer` as input variables.
template = (
    "Combine the chat history, previous answer, and follow up question into "
    "a standalone question. Chat History: {chat_history} "
    "Previous answer: {previous_answer} "
    "Follow up question: {question}"
)
prompt = PromptTemplate.from_template(template)

In this template, {previous_answer} is a placeholder for the previous answer generated by the LLM. You need to modify the question_generator chain to pass the previous answer as an input variable.

Please note that this modification requires you to keep track of the previous answer and pass it as an input to the question_generator chain. This might require further modifications to your code depending on how you have structured your application.

I hope this helps! If you have any further questions, feel free to ask.

Best,
Dosu

Sources

• ConversationalRetrievalChain did not look into chat histroy while making an answer
• libs/langchain/langchain/chains/conversational_retrieval/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.