Understanding collapse_prompt in the map_reduce load_qa_chain in ConversationalRetrievalChain

[exfalsoquodlibet]

In the context of a ConversationalRetrievalChain, when using chain_type = "map_reduce",
I am unsure how collapse_prompt should be set up.

I understand that:

1. collapse_prompt is the prompt of the (optional) collapse_document_chain within a MapReduceDocumentsChain, which is the type of combined_docs_chain/load_qa_chain if chain_type is set to "map_reduce", and that collapse_document_chain is defined as "Chain to use to collapse intermediary results if needed" see source code.

2. collapse_prompt is an optional argument, and that if not provided, no collapse_chain gets executed (source code):

if collapse_prompt is None:
        collapse_chain = None

3. If instead collapse_prompt is provided, then collapse_chain is executed and is a StuffDocumentChain (source code)

else:
        _collapse_llm = collapse_llm or llm
        collapse_chain = StuffDocumentsChain(
                    llm_chain=LLMChain(
                        llm=_collapse_llm,
                        prompt=collapse_prompt,
                        verbose=verbose,
                        callback_manager=callback_manager,
                        callbacks=callbacks,
                    )

Looking at the source code for StaffDocumentsChain, it looks like the default prompt is just to retrieve the whole content of the provided text:

def _get_default_document_prompt() -> PromptTemplate:
    return PromptTemplate(input_variables=["page_content"], template="{page_content}")


class StuffDocumentsChain(BaseCombineDocumentsChain):
    """Chain that combines documents by stuffing into context."""

    llm_chain: LLMChain
    """LLM wrapper to use after formatting documents."""
    document_prompt: BasePromptTemplate = Field(
        default_factory=_get_default_document_prompt
    )
    """Prompt to use to format each document."""

Grateful for any clarifications on how the collapse_prompt should be crafted? Tnx!

[brett-matson]

@exfalsoquodlibet I'm wondering this too. Did you figure it out?

I'm trying to create a ConversationalRetrievalChain that can stream it's final answer back, but there's not a lot of information on the optional arguments that enable you to use a different LLM for the collapse stage.