How to introduce custom SystemMessagePromptTemplate inside ConversationalRetrievalChain while using StuffDocuments

[DLOVRIC2]

Hey everyone,

Hopefully, someone can help clarify the following problem.

I want to be able to do the following:

1. Provide a system message to prime the llm
2. Retrieve documents and call stuff documents chain on those
3. Call the conversational retrieval chain and run it to get an answer.

The following code examples are gathered through the Langchain python documentation and docstrings on some of their classes.

# Just loading some basic document into the vectorstore
loader = WebBaseLoader("https://lilianweng.github.io/posts/2023-06-23-agent/")
data = loader.load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)
all_splits = text_splitter.split_documents(data)
vectorstore = Chroma.from_documents(documents=all_splits, embedding=OpenAIEmbeddings())
retriever = vectorstore.as_retriever()


# Choosing the llm and memory
llm = ChatOpenAI(temperature=0)
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)


# --- The problem begins here:

# Stuff Document Chain
document_prompt = PromptTemplate(
    input_variables=["page_content"],
        template="{page_content}"
)
document_variable_name = "context"
prompt = PromptTemplate.from_template(
    "{context}"
)
llm_chain = LLMChain(llm=llm, prompt=prompt)
combine_docs_chain = StuffDocumentsChain(
    llm_chain=llm_chain,
    document_prompt=document_prompt,
    document_variable_name=document_variable_name
)


# Conversation retrieval chain
template = (
    "Combine the chat history and follow up question into "
    "a standalone question. Chat History: {chat_history}"
    "Follow up question: {question}"
)
prompt = PromptTemplate.from_template(template)
question_generator_chain = LLMChain(llm=llm, prompt=prompt)
chain = ConversationalRetrievalChain(
    combine_docs_chain=combine_docs_chain,
    retriever=retriever,
    question_generator=question_generator_chain,
    memory=memory,
)

The code provided above works completely fine, but I'm struggling to find a place where I can add a system message to prime the chatbot in a certain way.

Usually, when not including documents to be retrieved from the vectorstore and I just want to have a conversation chain,
I would do something like this:

llm = ChatOpenAI(temperature=0)
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
messages = [
    SystemMessagePromptTemplate.from_template("You are a helpful assistant."),
    MessagesPlaceholder(variable_name="chat_history"),
    HumanMessagePromptTemplate.from_template("{question}")
]
prompt = ChatPromptTemplate.from_messages(messages=messages)

chain = LLMChain(
    llm=llm,
    prompt=prompt,
    verbose=True,
    memory=memory
)

print(chain({"question": "Hello"}))

So my question is this:
"How do I combine the prompt_template that I have defined in the second example (which provides a system message to the llm) but add on top of that, the ability to stuff documents and send the stuff documents to the main conversation chain?"

The ideal structure I am looking to have is:

• prompt (from messages)
• stuff documents chain
• ConversationalRetrievalChain

Any help would be appreciated, thanks!

[dosubot[bot]]

🤖

Hello,

Based on your question, it seems you want to introduce a custom SystemMessagePromptTemplate into the ConversationalRetrievalChain while using StuffDocuments. You can achieve this by creating a new ChatPromptTemplate and adding your custom SystemMessagePromptTemplate to it. Then, you can use this ChatPromptTemplate to format your messages.

Here is an example of how you can do this:

from langchain.prompts import SystemMessagePromptTemplate, ChatPromptTemplate

# Create a SystemMessagePromptTemplate
system_message_template = SystemMessagePromptTemplate.from_template("Your custom system message here")

# Create a ChatPromptTemplate and add the system message template to it
chat_template = ChatPromptTemplate.from_messages([system_message_template])

# Now you can use this chat_template to format your messages
messages = chat_template.format_messages()

In this example, SystemMessagePromptTemplate.from_template("Your custom system message here") creates a new SystemMessagePromptTemplate with your custom system message. ChatPromptTemplate.from_messages([system_message_template]) creates a new ChatPromptTemplate and adds your custom SystemMessagePromptTemplate to it. Finally, chat_template.format_messages() formats your messages according to the templates in the ChatPromptTemplate.

Please replace "Your custom system message here" with your actual system message.

In the context shared, the ChatPromptTemplate.from_messages([message]) method is used to create a new ChatPromptTemplate from a list of messages. This method validates the input variables of the messages and raises a ValueError if there is a mismatch between the expected and actual input variables. Therefore, ensure that the input variables of your custom SystemMessagePromptTemplate match the expected input variables.

Also, in the context shared, it's important to note that the BasePromptTemplate class has a validate_variable_names method that checks if the variable names include restricted names such as 'stop'. If they do, a ValueError is raised. Therefore, ensure that your custom SystemMessagePromptTemplate does not include these restricted names.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• Question: How can I include SystemMessage with RetrievalQA
• libs/langchain/langchain/prompts/chat.py
• libs/langchain/langchain/schema/prompt_template.py
• libs/langchain/langchain/chains/combine_documents/stuff.py
• libs/langchain/langchain/chains/conversational_retrieval/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[DLOVRIC2]

@dosu-bot how would I integrate that custom SystemMessagePromptTemplate with StuffDocumentChain and ConversationalRetrievalChain?

[dosubot[bot]]

🤖

Hello,

To integrate your custom SystemMessagePromptTemplate and ChatPromptTemplate with StuffDocumentChain and ConversationalRetrievalChain, you can follow the steps below:

1. Create your custom SystemMessagePromptTemplate and ChatPromptTemplate using the provided classes in the LangChain framework. You can use the from_template class method to create instances of these classes.

system_message_template = SystemMessagePromptTemplate.from_template("Your system message template here")
chat_message_template = ChatPromptTemplate.from_template("Your chat message template here")

2. Use the format_messages method of the ChatPromptTemplate class to format your messages. This method takes keyword arguments that correspond to the variables in your template.

messages = chat_message_template.format_messages(name="Bob", user_input="What is your name?")

3. Pass these formatted messages to your StuffDocumentChain and ConversationalRetrievalChain instances. The exact way to do this will depend on the specific methods and interfaces provided by these classes, which are not included in the provided context.

In the context shared, the from_template method of the ChatPromptTemplate class creates a chat template consisting of a single message assumed to be from the human. If you want to create a chat template with multiple messages, you can use the from_messages method instead.

Also, the ConditionalPromptSelector class is used to select between different prompts based on certain conditions. In this case, it selects between the default PROMPT and the CHAT_PROMPT based on whether the model is a chat model. You can use this class to select between your custom SystemMessagePromptTemplate and ChatPromptTemplate based on your own conditions.

In the context shared, the BasePromptTemplate class has a format method that you can use to format your custom templates. This method takes keyword arguments that correspond to the variables in your template. You can use this method to format your SystemMessagePromptTemplate and ChatPromptTemplate.

formatted_system_message = system_message_template.format(variable1="value1", variable2="value2")
formatted_chat_message = chat_message_template.format(variable1="value1", variable2="value2")

The BasePromptTemplate class also has a save method that you can use to save your custom templates to a file. This can be useful if you want to reuse your templates in the future.

system_message_template.save(file_path="path/to/system_message_template.yaml")
chat_message_template.save(file_path="path/to/chat_message_template.yaml")

The format_document function can be used to format a Document instance based on a BasePromptTemplate instance. This function takes a Document instance and a BasePromptTemplate instance as arguments and returns a formatted string. You can use this function to format your Document instances based on your custom SystemMessagePromptTemplate and ChatPromptTemplate.

from langchain.schema import Document
doc = Document(page_content="This is a joke", metadata={"page": "1"})
formatted_system_message = format_document(doc, system_message_template)
formatted_chat_message = format_document(doc, chat_message_template)

Please note that the exact way to pass these formatted messages to your StuffDocumentChain and ConversationalRetrievalChain instances will depend on the specific methods and interfaces provided by these classes, which are not included in the provided context.

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

• Question: How can I include SystemMessage with RetrievalQA
• libs/langchain/langchain/prompts/chat.py
• libs/langchain/langchain/chains/question_answering/stuff_prompt.py
• libs/langchain/langchain/schema/prompt_template.py
• libs/langchain/tests/unit_tests/prompts/test_chat.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Fizunimo]

This may have already been resolved, but here is an example from my environment.
After creating the prompt as follows,

llm = ChatOpenAI(temperature=0)
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
messages = [
    SystemMessagePromptTemplate.from_template("You are a helpful assistant."),
    MessagesPlaceholder(variable_name="chat_history"),
    HumanMessagePromptTemplate.from_template("{question}")
]
prompt = ChatPromptTemplate.from_messages(messages=messages)

Using this prompt for the llm_chain prompt for stuff document chain worked.
The prompts should be changed to suit your intended use.

Stuff Document Chain

document_prompt = PromptTemplate(
    input_variables=["page_content"],
        template="{page_content}"
)

llm_chain = LLMChain(llm=llm, prompt=prompt)
combine_docs_chain = StuffDocumentsChain(
    llm_chain=llm_chain,
    document_prompt=document_prompt,
    document_variable_name=document_variable_name
)

[DLOVRIC2]

Thanks @Fizunimo this was helpful.

I also found this issue relevant #6741