Bug in create_react_agent with response_format??

[Courvoisier13]

Hi,
I have been getting an issue with create_react_agent with response_format enabled. The issue occurs when I use create_react_agent with no tools and a prompt and a response_format:

        """Worker to route to next. If no workers needed, route to FINISH."""
        next: Literal["option 1", "option 2", "option 3", "option 4"]
agent = create_react_agent(
        llm,
        tools=[],
        prompt=system_prompt,
        response_format=Router
    )
response = agent.invoke({"messages": messages})

The prompt asks the agent to pick one of ENUMs in next.
I am getting a response with a message say equal to "option 1", by virtue of the prompt, it restricts the answer to one of the ENUMs in the Router.
However, in the response, I get a structured_response equal to "option 4". I thought it was weird that the llm generate an exact answer equal to one of the ENUMs but the parsed one is something else.

I debugged and found that two calls were happening in chat_agent_executor.py in langgraph/prebuilt. The first to call_model which gives the right answer and the other to generate_structured_response which gives the structured_response field. After following the trail of generate_structured_response I reach _create_chat_result in langchain_openai/chat_models/base.py. However, There is no prompt anymore. The llm is simply getting the messages, with no prompt, and it's just picking that makes no sense.

Adding the system prompt to the front of the messages solves the issue. But somewhere along the way the system prompt given to the llm when using response_format disappear from the final llm call.

This seems like a bug, doesn't it?

[vbarda]

it's not a bug, but i can see how this can be confusing - you can pass dedicated prompt for the structured output LLM call by passing a tuple (prompt, Schema) to response_format -- see API reference https://langchain-ai.github.io/langgraph/reference/prebuilt/#langgraph.prebuilt.chat_agent_executor.create_react_agent

[Courvoisier13]

If I already have a prompt set in create_react_agent why would I need to add another one in the response_format? This is not explained at all in the doc.
it says "The prompt will be used together with the model that is being used to generate the structured response." but shouldn't the prompt argument be part of the model config already? It makes it sound like if I added a prompt in the response_format I would just be duplicating.

[vbarda]

they are separate because the LLM calls in the react loop are different from the LLM call for structured output - it's a separate LLM call after the react loop is done

for example, you might want to provide specific instructions for structured output that make no sense for the main system prompt for the agent

[Courvoisier13]

Thanks, it would be great to update the documentary to make it clear that the main prompt is not used at all in the structured output. because currently this is not clear at all.

[HelloThisIsFlo]

Agreed! @Courvoisier13

While I appreciate the approach @vbarda, I also just spend 30min in deep debugging to figure out why I have 2 LLM calls, with one seemingly missing the system prompt.

It is a clever approach you have here, but I think we won't be the only two confused here. Especially when people coming from ADK, Agno, and such expect structured output to just be passed to the model.

Maybe adding a guide dedicated to how structured output works for the react agent could be helpful. I'm not sure. Or maybe in the doc, turning the Note into a orange Warning or something that really catch the attention.
It's nitpicky, but Langgraph is a framework for nitpickers anyway 😁

Awesome tool btw, and thanks for the explanation ♥️