ChatOpenAI: simultaneous use of with_structured_output and bind_tools

[jcampbell]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

I would like to be able to use ChatOpenAI to provide both tools and a response_format schema. The OpenAI API allows this directly by simply providing both tools and response format. It seems to provide a good mechanism for ensuring structured responses--especially taking advantage of constrained decoding--that both expect results from tool calls and do not.

Currently this seems to be possible if I manually bind result format and tools to the model, but that looses the helper functionality provided by the with_structured_output and bind_tools helpers.

https://openai.com/index/introducing-structured-outputs-in-the-api/

Motivation

I would like to directly take advantage of the model's choice of whether to call a tool or not based on the context, while still ensuring that if a tool call is not necessary the resulting message is structured.

Proposal (If applicable)

I think an ideal option would be to allow with_structured_output to accept a tools parameter, which, if provided, first binds the tools and then binds the response format, and builds the appropriate parser chain.

[jcampbell]

Consider this message transcript as a motivating example:

{
  "model": "gpt-4o-2024-08-06",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful math tutor. You must use a calculator to check that the value you provide for x satisfies the equation before providing a final answer."
    },
    {
      "role": "user",
      "content": "solve x + 4 = 20"
    },
		{
				"role": "assistant",
				"content": null,
				"tool_calls": [
					{
						"id": "call_hpE6j6NcSXYtsPxPwoGhNk30",
						"type": "function",
						"function": {
							"name": "calculator",
							"arguments": "{\"x\":20,\"y\":4,\"op\":\"-\"}"
						}
					}
				]
		},
		{
			"role": "tool",
			"tool_call_id": "call_hpE6j6NcSXYtsPxPwoGhNk30",
			"name": "calculator",
			"content": "16"
		}
		
  ],
	"tools": [
    {
      "type": "function",
      "function": {
        "name": "calculator",
        "description": "Calculate arithmetic expressions for two operands.",
        "strict": true,
        "parameters": {
          "type": "object",
          "properties": {
            "x": {
              "type": "number"
            },
						"y": {
              "type": "number"
            },
						"op": {
              "type": "string",
							"enum": ["+", "-", "*", "/"]
            }
          },
          "required": ["x", "y", "op"],
          "additionalProperties": false
        }
      }
    }
  ],
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "math_response",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "steps": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "explanation": {
                  "type": "string"
                },
                "output": {
                  "type": "string"
                }
              },
              "required": ["explanation", "output"],
              "additionalProperties": false
            }
          },
          "final_answer": {
            "type": "string"
          }
        },
        "required": ["steps", "final_answer"],
        "additionalProperties": false
      }
    }
  }
}

[jcampbell]

Confirmed that this approach works! Thank you.

[kapis]

That doesn't seem to work anymore :(
Using openai==1.59.3 langchain_openai==0.2.14

[anmol-aidora]

I tried but it is not giving correct responses every time:
For first question (unnecessary details omitted) (no tool calls):

content='{"response":"The color of a white rabbit is typically white, which means it has a coat that is predominantly white in color. However, some white rabbits may have slight variations in shade or markings, but they are generally recognized for their pure white fur." ,"custom_field":"Color description of a white rabbit."}' 
additional_kwargs={'parsed': GenericResponse(response='...', custom_field='Color description of a white rabbit.')}

For second question (no parsed formatted output):

content=''
tool_calls=[{'name': 'get_current_datetime', 'args': {'timezone_str': 'UTC'}, 'id': 'call_MKElt4rZTzpIJLpxxtFEPEYV', 'type': 'tool_call'}] 

Seems like it can do only one thing at a time.

Thoughts?

[deflagg]

I have not tried this since Oct 2024. Step through the LangChain code to investigate if it is meeting all of OpenAI's structured output requirements. Step through bind_tools method of the ChatOpenAI object. Research what it's doing and make sure it adheres to the OpenAI API specifications. This is what I did to find a workaround.

llm.bind_tools([get_current_datetime], strict=True, response_format=GenericResponse)

Second Option: Use Cursor or VS Code with CoPilot to analyze the codefile where ChatOpenAI.bind_tools lives. Ask it why the shit ain't working

[deflagg]

I've been using AgentSDK from OpenAI until there's a fix. I starting to find it easier to use. The documentation is better. If Langchain and LangGraph delete all their documentation and start over, that would help out alot.

[radsimu-spektr]

hello langchain. I want to use openai to perform a web search using their web search tool and have the response in a particular json format. This works with vanial openai client and their web playground. In langchain, the most natural way you would expect to use this is to chain .bind_tools with a .with_structured_output. This is not allowed, at least not in the js version of langchain

[radsimu-spektr]

I think I found a solution, by using the .bind which works with any runnable. It is not very pretty, but seems to work.

The only issue here is that typescript complains that 'tools' does not exist in type 'Partial<RunnableConfig<Record<string, any>>>'. If I declare llm as any works to bypass that typecheck but usually any is not recommended..

    .withStructuredOutput(outputSchema)
    .bind({
          tools: [
            {
              type: 'web_search_preview',
              search_context_size: 'medium',
            },
          ],
          tool_choice: {
            type: 'web_search_preview',
          },
        });
      }

[nickaein]

The official documentation recommends the following pattern for structured outputs with tools:

model_with_tools = model.bind_tools([tool1, tool2])
structured_model = model_with_tools.with_structured_output(schema)

But this doesn't work for web_search tool of OpenAI in Python SDK.

After some experimentation, this is the cleanest way that I got it working in Python:

llm = llm.with_structured_output(
    schema=SchemaClassInPydantic,
    strict=True,
    include_raw=True,
    tools=[{"type": "web_search_preview"}],
)

This same approach may work in the TypeScript SDK.