Responses API: Structured Output Models & Tool Calls are Mutually Exclusive

[nahiluhmot]

The Responses API resource works when specifying either a structured output model or a list of tool call models, but it does not support both specified at once. After a quick look at the client, this appears to be happening because Responses#get_structured_output_models assumes that these parameters are mutually exclusive.

Repro Script

This repro script is a modification of the structured_outputs_responses.rb example which adds a function tool.

require "openai"

class Location < OpenAI::BaseModel
  required :address, String
  required :city, String, doc: "City name"
  required :postal_code, String, nil?: true
end

# Participant model with an optional last_name and an enum for status
class Participant < OpenAI::BaseModel
  required :first_name, String
  required :last_name, String, nil?: true
  required :status, OpenAI::EnumOf[:confirmed, :unconfirmed, :tentative]
end

# CalendarEvent model with a list of participants.
class CalendarEvent < OpenAI::BaseModel
  required :name, String
  required :date, String
  required :participants, OpenAI::ArrayOf[Participant]
  required :optional_participants, OpenAI::ArrayOf[Participant, doc: "who might not show up"], nil?: true
  required :is_virtual, OpenAI::Boolean
  required :location,
           OpenAI::UnionOf[String, Location],
           nil?: true,
           doc: "Event location"
end

class LookupCalendar < OpenAI::BaseModel
  required :first_name, String
  required :last_name, String
end

client = OpenAI::Client.new

response = client.responses.create(
  model: "gpt-4o-2024-08-06",
  input: [
    {role: :system, content: "Extract the event information."},
    {
      role: :user,
      content: <<~CONTENT
        Alice Shah and Lena are going to a science fair on Friday at 123 Main St. in San Diego.
        They have also invited Jasper Vellani and Talia Groves - Jasper has not responded and Talia said she is thinking about it.
      CONTENT
    }
  ],
  tools: [LookupCalendar],
  text: CalendarEvent
)

response
  .output
  .flat_map { _1.content }
  # filter out refusal responses
  .grep_v(OpenAI::Models::Responses::ResponseOutputRefusal)
  .each do |content|
    # parsed is an instance of `CalendarEvent`
    pp(content.parsed || content)
  end

Output:

$ ruby ./script.rb
/Users/REDACTED/.rbenv/versions/3.4.3/lib/ruby/gems/3.4.0/gems/openai-0.25.0/lib/openai/internal/transport/base_client.rb:410:in 'OpenAI::Internal::Transport::BaseClient#send_request': {url: "https://api.openai.com/v1/responses", status: 400, body: {error: {message: "Invalid type for 'tools[0]': expected an object, but got a string instead.", type: "invalid_request_error", param: "tools[0]", code: "invalid_type"}}} (OpenAI::Errors::BadRequestError)

If you comment out the tools key-value pair from the responses.create call and re-run the script, it works. Similarly, if you comment out the text key-value pair and re-run the script, it works.

I would expect the API client to handle both structured outputs and tool calls in the same API call; please let me know if that's not the case.

[ms-jpq]

@renfredxh lets figure this out tomorrow 👍