Langchain MRKL Agent not giving useful Final Answer

[aju22]

Here is the code I'm using for initializing a Zero Shot ReAct Agent with some tools for fetching relevant documents from a vector database:

chat_model = ChatOpenAI(
            model_name="gpt-3.5-turbo",
            temperature="0",
            openai_api_key=openai_api_key,
            streaming=True,
            # verbose=True)

llm_chain = LLMChain(llm=chat_model, prompt=prompt)

agent = ZeroShotAgent(llm_chain=llm_chain, tools=tools, verbose=True, handle_parsing_errors=True)
agent_chain = AgentExecutor.from_agent_and_tools(
    agent=agent, tools=tools, verbose=True, memory=memory
)

However when I query for a response.

query = "Can you explain a use case example of chain of thought prompting in detail?"
res = agent_chain(query)

This is the response I get back:

> Entering new  chain...
Thought: The question is asking for a detailed explanation of a use example of chain-of-thought prompting.
Action: Lookup from database
Action Input: "use example of chain-of-thought prompting"
Observation: Sure! Here's an example of chain-of-thought prompting:

Let's say we have a language model that is trained to solve math word problems. We want to use chain-of-thought prompting to improve its reasoning abilities.

The prompt consists of triples: input, chain of thought, output. For example:

Input: "John has 5 apples."
Chain of Thought: "If John gives 2 apples to Mary, how many apples does John have left?"
Output: "John has 3 apples left."

In this example, the chain of thought is a series of intermediate reasoning steps that lead to the final output. It helps the language model understand the problem and perform the necessary calculations.

By providing these chain-of-thought exemplars during training, the language model learns to reason step-by-step and can generate similar chains of thought when faced with similar problems during inference.

This approach of chain-of-thought prompting has been shown to improve the performance of language models on various reasoning tasks, including arithmetic, commonsense, and symbolic reasoning. It allows the models to decompose complex problems into manageable steps and allocate additional computation when needed.

Overall, chain-of-thought prompting enhances the reasoning abilities of large language models and helps them achieve state-of-the-art performance on challenging tasks.
Thought:I have provided a detailed explanation and example of chain-of-thought prompting.
Final Answer: Chain-of-thought prompting is a method used to improve the reasoning abilities of large language models by providing demonstrations of chain-of-thought reasoning as exemplars in prompting. It involves breaking down multi-step problems into manageable intermediate steps, leading to more effective reasoning and problem-solving. An example of chain-of-thought prompting is providing a language model with a math word problem prompt consisting of an input, chain of thought, and output. By training the model with these exemplars, it learns to reason step-by-step and can generate similar chains of thought when faced with similar problems during inference. This approach has been shown to enhance the performance of language models on various reasoning tasks.

> Finished chain.

As you can observe, The model has a very thorough and exact answer in it's observation. However in the next thought, the model thinks it is done providing a detailed explanation and example to the human. So the final answer is just some basic information, not really answering the question in necessary detail.

I feel like somewhere in the intermediate steps, the agent thinks it has already answered to the human, and hence just does not bother to give that as the final answer.

Can someone please help me figure out, how can I make the model output it's observation as the final answer. Or to stop making the model assume it has already answered the question to the human.

Will playing around with the prompt template work?

[dosubot[bot]]

Answer generated by a 🤖

Answer

I understand that you're experiencing an issue where the LangChain MRKL Agent is not providing a detailed final answer as expected. It seems that the agent is generating a comprehensive response during the observation phase, but the final answer is not reflecting this level of detail.

The issue appears to be related to how the final answer is being generated within the LangChain framework. The final answer isn't directly taken from the observation, but is instead produced by the model based on the thoughts and observations it has made during the process.

One potential solution could be to modify the prompt template that guides the model's responses. Currently, your template may be leading the model to believe it has sufficiently answered the question after making the observation, hence the less detailed final answer.

Here's a suggestion on how you might adjust the prompt template:

FORMAT_INSTRUCTIONS = """Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer based on my observation
Final Answer: the final answer to the original input question is the detailed explanation from the observation"""

In this revised template, the model is instructed to consider the detailed explanation from the observation as the final answer. This could potentially result in a more detailed final answer that better addresses your query.

Please note that this is just one potential solution and it might not work perfectly in all scenarios. If you believe this is a bug that could impact other users, feel free to make a pull request with your proposed changes.

If this solution doesn't align with your experience, I would recommend ensuring that you're using the latest version of LangChain, as updates often include bug fixes and improvements that could potentially resolve your issue.

Sources

• langchain/agents/mrkl/prompt.py

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

[aju22]

Yes thank you! It seems like this may work. However I keep getting OutputParserException: Could not parse LLM output.

I have tried setting handle_parsing_errors=True as well as handle_parsing_errors="Check your output and make sure it conforms!", and yet most of the times I find myself getting the OutputParserException.

Here is an example of the error:

> Entering new  chain...
Thought: The question is asking for a detailed explanation of a use example of chain of thought prompting. I should first check if there is a clear answer in the database.

Action: Lookup from database
Action Input: "use example of chain of thought prompting"

Observation: Sure! Here's an example of chain-of-thought prompting:

Let's say we have a language model that needs to solve a math word problem. The problem is: "John has 5 apples. He gives 2 apples to Mary. How many apples does John have now?"

With chain-of-thought prompting, we provide the model with a prompt that consists of triples: input, chain of thought, output. In this case, the prompt could be:

Input: "John has 5 apples. He gives 2 apples to Mary."
Chain of Thought: "To solve this problem, we need to subtract the number of apples John gave to Mary from the total number of apples John had."
Output: "John now has 3 apples."

By providing the model with this chain of thought, we guide it through the reasoning process step-by-step. The model can then generate the correct answer by following the provided chain of thought.

This approach of chain-of-thought prompting helps the language model to decompose multi-step problems into intermediate steps, allowing for better reasoning and problem-solving abilities.
Thought:
---------------------------------------------------------------------------
OutputParserException                     Traceback (most recent call last)
[<ipython-input-76-951eb95eb01c>](https://localhost:8080/#) in <cell line: 2>()
      1 query = "Can you explain a use example of chain of thought prompting in detail?"
----> 2 res = agent_chain(query)

6 frames
[/usr/local/lib/python3.10/dist-packages/langchain/agents/mrkl/output_parser.py](https://localhost:8080/#) in parse(self, text)
     40 
     41         if not re.search(r"Action\s*\d*\s*:[\s]*(.*?)", text, re.DOTALL):
---> 42             raise OutputParserException(
     43                 f"Could not parse LLM output: `{text}`",
     44                 observation="Invalid Format: Missing 'Action:' after 'Thought:'",

OutputParserException: Could not parse LLM output: `I have found a clear answer in the database that explains a use example of chain of thought prompting.`

Seems like it's final thought is not "Thought: I now know the final answer based on my observation"
Is there any other way in which I can mitigate this problem to get consistent outputs?