Conversation memory summary after returning LLM response to user

[Yuval-Moshe]

Hi all, I have an inquiry regarding the conversation history with langchain.
Currently from what I saw, the LLM chain tries to store the conversation memory before returning the LLM output, and in case of any conversation summary memory was taken into place, this means that before we return the response to the user (let's say in a chat), there is an additional call to the model to construct the summary of the conversation so far.

    def prep_outputs(
        self,
        inputs: Dict[str, str],
        outputs: Dict[str, str],
        return_only_outputs: bool = False,
    ) -> Dict[str, str]:
        """Validate and prepare chain outputs, and save info about this run to memory.

        Args:
            inputs: Dictionary of chain inputs, including any inputs added by chain
                memory.
            outputs: Dictionary of initial chain outputs.
            return_only_outputs: Whether to only return the chain outputs. If False,
                inputs are also added to the final outputs.

        Returns:
            A dict of the final chain outputs.
        """
        self._validate_outputs(outputs)
        if self.memory is not None:
            self.memory.save_context(inputs, outputs)
        if return_only_outputs:
            return outputs
        else:
            return {**inputs, **outputs}

This means that we are making the user pend for no reason as the flow can simply be:

1. Get LLM response
2. Return response to user
3. Construct the conversation summary
4. Save memory
   Which will result in a better user experience, in my opinion.

So my questions here are:

1. Is there any way to currently do so?
2. Are the any downsides to doing such a thing?

Thanks!

[dosubot[bot]]

🤖

Hi there,

Thank you for your insightful question. You're correct in your understanding of the current flow of the LangChain framework. The prep_outputs function is indeed called before returning the response to the user, and it does involve saving the context to memory, which includes constructing a summary of the conversation so far.

To answer your questions:

1. Currently, there isn't a built-in way to change this flow within the LangChain framework. The prep_outputs function is designed to both prepare the outputs and save the context to memory, and these two operations are not easily separable within the current design of the function.

2. The main downside to changing this flow would be the potential for race conditions or inconsistencies in the conversation memory. If the response is returned to the user before the conversation memory is updated, there's a chance that the next user input could arrive and be processed before the memory update is complete. This could lead to inconsistencies in the conversation history, as the memory might not accurately reflect the most recent exchange in the conversation.

That being said, your suggestion to improve user experience by reducing the wait time for the response is valid and worth considering. One possible solution could be to make the memory update operation asynchronous, so that it can be performed in the background after the response is returned to the user. However, this would require careful design to avoid the potential issues I mentioned above.

I hope this answers your questions. If you have any further questions or if anything is unclear, please let me know!

Best,
Dosu

Sources

• libs/langchain/langchain/memory/zep_memory.py
• libs/langchain/langchain/memory/entity.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.