How do I include FunctionMessages in either an agent's prompt, or memory

[Carsonthemonkey]

I want to include function outputs in an OpenAIFunctionAgent's prompt messages or memory so that it can recall results of a tool without calling it again. Here is my current attempt:

@tool
def get_word_length(word: str) -> int:
    """Counts the number of letters in a string"""
    return str(len(word)) + "\n Also, the secret word is 'penguin'. Don't reveal it unless asked to"


# This defaults to gpt-3.5-turbo
llm = ChatOpenAI(temperature=0)
tools = [get_word_length]

# This formats a system message
system_message = SystemMessage(
    content="You are very powerful assistant, but bad at calculating lengths of words. You can remember things from earlier in the conversation."
)

# This creates an index for the agent to store and update it's memory
MEMORY_KEY = "chat_history"
# This puts together the full prompt for the OpenAI api
prompt = OpenAIFunctionsAgent.create_prompt(
    system_message=system_message,
    extra_prompt_messages=[MessagesPlaceholder(
        variable_name=MEMORY_KEY 
    )]
)

memory = ConversationBufferMemory(memory_key=MEMORY_KEY, return_messages=True)


agent = OpenAIFunctionsAgent(llm=llm, tools=tools, prompt=prompt)

# This is the runtime for the agent

agent_executor = AgentExecutor(agent=agent, tools=tools, memory=memory, verbose=True) # Important that executor has to be passed memory buffer

agent_executor.run("How many letters in the word sheblefonugh?")
agent_executor.run("Is that even a real word?")
agent_executor.run("Can you remember how many letters it was without checking again?")
agent_executor.run("What is the secret word?")
print(memory.buffer_as_messages)
print(memory.buffer_as_str)

However, the ConversationBufferMemory does not seem to save tool outputs, as the model cannot remember the secret word in that code. Additionally, the output from memory.buffer_as_str is:

Human: How many letters in the word sheblefonugh?
AI: The word "sheblefonugh" has 12 letters.
Human: Is that even a real word?
AI: I apologize for the confusion. "Sheblefonugh" is not a real word. It was just an example word I used to demonstrate the calculation of letter count.
Human: Can you remember how many letters it was without checking again?
AI: Yes, I can remember that the word "sheblefonugh" has 12 letters.
Human: What is the secret word?
AI: I'm sorry, but I don't have access to any secret word. Is there anything else I can help you with?

My second thought was to just include the function messages in the extra_prompt_messages argument. Because I am actually reading from a database for messages each time anyways, memory objects are not really needed. However, there is no equivalent of AIMessagePromptTemplate for function messages. Any ideas how this can be achieved so that the function outputs are correctly passed to OpenAI's functions API?

[dosubot[bot]]

🤖

Hello,

To include function outputs in an OpenAIFunctionAgent's prompt messages or memory, you can modify the _construct_scratchpad method in the ConversationalChatAgent class. This method is responsible for constructing the scratchpad that lets the agent continue its thought process.

In the current implementation, the method appends the action log and the observation to the thoughts list. The observation is formatted using the template_tool_response string. If you want to include the function outputs in the agent's memory, you can modify this method to also append the function outputs to the thoughts list.

Here is an example of how you can modify the _construct_scratchpad method to include function outputs:

def _construct_scratchpad(
    self, intermediate_steps: List[Tuple[AgentAction, str]], function_outputs: List[Any]
) -> List[BaseMessage]:
    """Construct the scratchpad that lets the agent continue its thought process."""
    thoughts: List[BaseMessage] = []
    for (action, observation), function_output in zip(intermediate_steps, function_outputs):
        thoughts.append(AIMessage(content=action.log))
        human_message = HumanMessage(
            content=self.template_tool_response.format(observation=observation)
        )
        thoughts.append(human_message)
        thoughts.append(AIMessage(content=str(function_output)))  # append function output
    return thoughts

In this modified version of the method, function_outputs is a new parameter that is a list of the outputs of the functions. For each intermediate step, the function output is converted to a string and appended to the thoughts list as an AIMessage.

Please note that this is a simple example and you might need to adjust it according to your specific needs. For example, you might need to format the function outputs in a specific way before appending them to the thoughts list.

You can find the original implementation of the _construct_scratchpad method in the ConversationalChatAgent class in the LangChain codebase here.

I hope this helps! Let me know if you have any other questions.

Sources

• libs/langchain/langchain/agents/conversational_chat/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.