Can I use the system prompt to specify the format in conversationalretrievalchain?

[Fizunimo]

I would like to specify the format for response generation in a system role.
The current specification is to give the reconstructed questions, materials and formatting instructions all at once, but can the formatting instructions be split into system roles?
Thank you in advance.

[dosubot[bot]]

🤖

Hello,

Yes, it is indeed possible to split the formatting instructions into system roles for response generation in ConversationalRetrievalChain. The formatting instructions are handled by the _get_chat_history function, which formats the chat history based on the role of the message sender. The roles are mapped in the _ROLE_MAP dictionary, where "human" is mapped to "Human: " and "ai" is mapped to "Assistant: ".

Here is the relevant code:

_ROLE_MAP = {"human": "Human: ", "ai": "Assistant: "}

def _get_chat_history(chat_history: List[CHAT_TURN_TYPE]) -> str:
    buffer = ""
    for dialogue_turn in chat_history:
        if isinstance(dialogue_turn, BaseMessage):
            role_prefix = _ROLE_MAP.get(dialogue_turn.type, f"{dialogue_turn.type}: ")
            buffer += f"\n{role_prefix}{dialogue_turn.content}"
        elif isinstance(dialogue_turn, tuple):
            human = "Human: " + dialogue_turn[0]
            ai = "Assistant: " + dialogue_turn[1]
            buffer += "\n" + "\n".join([human, ai])
        else:
            raise ValueError(
                f"Unsupported chat history format: {type(dialogue_turn)}."
                f" Full chat history: {chat_history} "
            )
    return buffer

This function iterates over the chat history and checks the type of each dialogue turn. If the dialogue turn is an instance of BaseMessage, it retrieves the role prefix from the _ROLE_MAP dictionary using the type attribute of the dialogue turn. If the dialogue turn is a tuple, it assumes the first element is the human's message and the second element is the AI's message. The function then concatenates the role prefix with the content of the message and adds it to the buffer. If the dialogue turn is neither a BaseMessage instance nor a tuple, it raises a ValueError. The function finally returns the buffer as a string, which represents the formatted chat history.

In the context shared, the STRUCTURED_FORMAT_INSTRUCTIONS, STRUCTURED_FORMAT_SIMPLE_INSTRUCTIONS, and PYDANTIC_FORMAT_INSTRUCTIONS constants in the format_instructions.py file provide additional formatting instructions for the assistant's responses. These instructions can be incorporated into the _get_chat_history function to further customize the formatting of the chat history.

You can find the relevant code in the following files:

• libs/langchain/langchain/chains/conversational_retrieval/base.py
• libs/langchain/langchain/agents/conversational_chat/prompt.py
• libs/langchain/langchain/output_parsers/format_instructions.py

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

• libs/langchain/langchain/chains/conversational_retrieval/base.py
• libs/langchain/langchain/agents/conversational_chat/prompt.py
• libs/langchain/langchain/output_parsers/format_instructions.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Fizunimo]

@dosu-bot
I just want the result of combining the sentence and prompt with combine_document without having to run llm.

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out. If you want to use the combine_document function to combine the sentence and prompt without running the Language Model (llm), you can modify the function to skip the self.initial_llm_chain.predict and self.refine_llm_chain.predict methods. Here's how you can do it:

def combine_docs(
    self,
    docs: List[Document],
    callbacks: Callbacks = None,
    **kwargs: Any,
) -> Tuple[str, dict]:
    """Combine by mapping first chain over all, then stuffing into final chain.

    Args:
        docs: List of documents to combine
        callbacks: Callbacks to be passed through
        **kwargs: additional parameters to be passed to LLM calls (like other
            input variables besides the documents)

    Returns:
        The first element returned is the single string output. The second
        element returned is a dictionary of other keys to return.
    """
    inputs = self._construct_initial_inputs(docs, **kwargs)
    # Skip the initial_llm_chain.predict method
    # res = self.initial_llm_chain.predict(callbacks=callbacks, **inputs)
    res = inputs
    refine_steps = [res]
    for doc in docs[1:]:
        base_inputs = self._construct_refine_inputs(doc, res)
        inputs = {**base_inputs, **kwargs}
        # Skip the refine_llm_chain.predict method
        # res = self.refine_llm_chain.predict(callbacks=callbacks, **inputs)
        res = inputs
        refine_steps.append(res)
    return self._construct_result(refine_steps, res)

Please note that this modification will skip the application of the initial_llm_chain.predict and refine_llm_chain.predict to each document, which might not be the intended behavior depending on your use case.

You can find more details about the combine_docs function in the RefineDocumentsChain class in the following files:

• libs/langchain/langchain/chains/combine_documents/map_reduce.py
• libs/langchain/langchain/chains/combine_documents/reduce.py
• libs/langchain/langchain/chains/combine_documents/stuff.py
• libs/langchain/langchain/chains/combine_documents/refine.py

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

• libs/langchain/langchain/chains/combine_documents/map_reduce.py
• libs/langchain/langchain/chains/combine_documents/reduce.py
• libs/langchain/langchain/chains/combine_documents/stuff.py
• libs/langchain/langchain/chains/combine_documents/refine.py
• libs/langchain/langchain/chains/llm.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Fizunimo]

@dosu-bot
I just want the result of combining the sentence and prompt with stuff combine_document without having to run llm.

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out with your question. To modify the combine_docs and acombine_docs methods in the StuffDocumentsChain class to combine a sentence and a prompt without running the self.llm_chain.predict and self.llm_chain.apredict methods, you can add a condition to check if the docs argument is a sentence and a prompt. If it is, you can simply return the combined sentence and prompt without calling the predict methods. Here's how you can do it:

def combine_docs(
    self, docs: List[Document], callbacks: Callbacks = None, **kwargs: Any
) -> Tuple[str, dict]:
    inputs = self._get_inputs(docs, **kwargs)
    # Check if docs is a sentence and a prompt
    if len(docs) == 2 and isinstance(docs[0], str) and isinstance(docs[1], str):
        # Combine the sentence and the prompt and return
        return docs[0] + ' ' + docs[1], {}
    else:
        # Call predict on the LLM.
        return self.llm_chain.predict(callbacks=callbacks, **inputs), {}

async def acombine_docs(
    self, docs: List[Document], callbacks: Callbacks = None, **kwargs: Any
) -> Tuple[str, dict]:
    inputs = self._get_inputs(docs, **kwargs)
    # Check if docs is a sentence and a prompt
    if len(docs) == 2 and isinstance(docs[0], str) and isinstance(docs[1], str):
        # Combine the sentence and the prompt and return
        return docs[0] + ' ' + docs[1], {}
    else:
        # Call predict on the LLM.
        return await self.llm_chain.apredict(callbacks=callbacks, **inputs), {}

This modification will ensure that if the docs argument is a sentence and a prompt, the combine_docs and acombine_docs methods will simply combine the sentence and the prompt and return, without calling the predict methods. In all other cases, the methods will behave as before.

Please note that the docs argument in the combine_docs and acombine_docs methods is a list of Document objects. If you want to pass a sentence and a prompt, you might need to convert them into Document objects first.

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

• libs/langchain/langchain/chains/combine_documents/stuff.py
• libs/langchain/langchain/chains/combine_documents/reduce.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.