Model AI chat events as a list of request/response messages, with each message containing a list of parts

[alexmojaki]

Area(s)

area:gen-ai

What's missing?

• As pointed out in GenAI handle tool message embedded within user message #1883, the current events don't match the API request structure when a message contains a combination of text and tool call responses.
• The naming and separation of events is messy and confusing:
  • There's different events for user and system messages, but the distinction is very artificial. The bodies have the same structure, the only difference is the role, which is also present in the body anyway, so a single event name would have worked.
  • genai: handle system role renamed to developer in openai #1877 shows that these roles change over time and aren't reliable enough to be embedded into the event name which needs to be very stable. One day new developers working with OpenAI will only be familiar with the role being called developer instead of system and won't understand why the event is called gen_ai.system.message.
  • Assistant message events can simultaneously contain text content with any number of tool calls, but for user messages these are split into multiple events. Why the inconsistency? Why not an event per tool call?
  • The event name gen_ai.tool.message doesn't make it clear that it means the result of a tool call, rather than the tool call itself. In other words, it's not clear at a glance whether it's sent by the user or assistant.
• There's no clear place for multi-modal content (Support for Multi modal inputs and generations #1556), e.g. a message containing both text and images.
• Tool calls have a type field which should apparently always be function, so its purpose is not clear.

Describe the solution you'd like

There needs to be a conceptual hierarchy, where a request consists of a list of messages, and a message consists of a list of 'parts'. Here's one way the events could look:

• One event per message in the request, each with the same event name, e.g. gen_ai.message.
• Each message event has role and content keys in the body, similar to the current events.
• role is required and is used to distinguish between user, system, and assistant messages.
• content in the body is an array of parts.
• Each part is an object with a type field. Some possible values for type are text, image, tool_call, and tool_response.
  • If needed and possible, this field should account for multiple different types of tool call that the existing type field seems to be meant for.
• The separate tool_calls array in the bodies of assistant and choice events are removed in favour of tool_call parts in the content array.
• User and assistant messages (including the response choice events) all have the same structure, except that the choice events have additional fields (index, finish_reason) that don't make sense in the request events.

Support for this data model:

• The structure of Google's Gemini/Vertex request messages: https://github.com/googleapis/googleapis/blob/58be301346758c9a342de5632c3f9284d05c4b95/google/cloud/aiplatform/v1/content.proto#L80-L100
• OpenAI uses an array of parts in each message for image and audio inputs: https://platform.openai.com/docs/api-reference/chat/create
• pydantic_ai has ModelMessage which is either ModelRequest or ModelResponse, each of which has a parts field which is a list of things like TextPart or ToolCallPart. This may seem like a biased example but the point is that pydantic_ai works with many different underlying AI APIs from different companies, and it translates between all those schemas and ModelMessage out of necessity.

[michaelsafyan]

It might be even easier from an instrumentation standpoint if the events aligned with "parts". That is, each "part" is mapped to a distinct event.

However, the format of the data/event may need to depend on the kind of part. For example, a "part" that is just a piece of text is going to have a different representation than a "part" that is an image.

This would also better fit within existing per-signal limits to split these up.

[alexmojaki]

Here's an example to sketch how this might look. With the current spec:

gen_ai.assistant.message {
  content: assistant response text
  tool_calls: [
    {
      id: call1,
      function: {
        name: function1,
        arguments: ...
      },
      type: function
    },
    {
      id: call2,
      function: {
        name: function2,
        arguments: ...
      },
      type: function
    }
  ]
}

gen_ai.user.message {
  content: here's the responses to those tool calls
}

gen_ai.tool.message {
  id: call1
  content: call1 results
}

gen_ai.tool.message {
  id: call2
  content: call2 results
}

gen_ai.choice {
  index: 0
  message: {
    content: assistant response text
    tool_calls: [
      {
        id: call3,
        function: {
          name: get_weather,
          arguments: ...
        },
        type: function
      }
    ]
  }
}

Here's the proposal using an array of parts in each message:

gen_ai.message {
  role: assistant
  content: [
    {
      type: text
      text: assistant response text
    },
    {
      type: tool_call,
      name: function1,
      arguments: ...
    },
    {
      type: tool_call,
      name: function2,
      arguments: ...
    }
  ]
}

gen_ai.message {
  role: user
  content: [
    {
      type: text
      text: here's the responses to those tool calls
    },
    {
      type: tool_response,
      id: call1
      content: call1 results
    },
    {
      type: tool_response,
      id: call2
      content: call2 results
    }
  ]
}

gen_ai.choice {
  index: 0
  content: [
    {
      type: text
      text: assistant response text
    },
    {
      type: tool_call,
      id: call3,
      name: get_weather,
      arguments: ...
    }
  ]
}

Here's with one event per part, using the event name as the discriminator instead of a type key:

gen_ai.message.part.text {
  role: assistant
  message_index: 0
  text: assistant response text
}

gen_ai.message.part.tool_call {
  role: assistant
  message_index: 0
  name: function1,
  arguments: ...
}

gen_ai.message.part.tool_call {
  role: assistant
  message_index: 0
  name: function2,
  arguments: ...
}

gen_ai.message.part.text {
  role: user
  message_index: 1
  text: here's the responses to those tool calls
}

gen_ai.message.part.tool_response {
  role: user
  message_index: 1
  id: call1
  content: call1 results
}

gen_ai.message.part.tool_response {
  role: user
  message_index: 1
  id: call2
  content: call2 results
}

gen_ai.choice.part.text {
   role: assistant
   choice_index: 0
   text: assistant response text
}

gen_ai.choice.part.tool_call {
  role: assistant
  choice_index: 0
  name: get_weather,
  arguments: ...
}

[michaelsafyan]

I prefer the last option, one event per part, with a slight refinement to the above:

Can message_index be made more general?

Perhaps something like:

field_path: response.candidates[0].parts[3]

[alexmojaki]

The idea of message_index is to be able to easily convert the flat representation into the 'array of parts' representation with simple 'group by' logic. These parts:

gen_ai.message.part.text {
  role: assistant
  message_index: 0
  text: assistant response text
}

gen_ai.message.part.tool_call {
  role: assistant
  message_index: 0
  name: function1,
  arguments: ...
}

have the same message_index so they are part of a single message. But they would have (sometimes very) different field_path values. In OpenAI the first would live at request.messages[0].content while the second would be at request.messages[0].tool_calls[0].

[alexmojaki]

From what I hear it sounds like people are in favour of this, specifically the flat version, which I prefer now too. I'd like to additionally request that all the top level keys have nice namespaced semconv-style names like gen_ai.tool_call.name instead of just name. This lets us make queries and storage more efficient.

[alexmojaki]

Looking at https://opentelemetry.io/docs/specs/semconv/general/events/#general-event-semantics, I see that body fields are usually supposed to be short and simple, not namespaced. However it also says:

If the field should be comparable across events with different EventName (or between an event and other telemetry items), it should be an attribute.

This suggests that basically all the fields discussed here should be attributes:

• All the events should have role
• All the request message part events should have message_index
• All the response choice part events should have choice_index
• Details of parts that vary between part types like text or image_url can mostly be generated by models (so they appear in choice events) and then those same generated values will reappear as messages in the request with role assistant.
• Tool call ID appears in both tool calls and tool responses, and is also useful in other telemetry for instrumenting the tool calls.
• Tool call name is not currently part of the definition for tool call responses, but IMO it should be there optionally. For example pydantic-ai allows populating this. It's also something that's useful in other telemetry, like tool call ID.

That leaves the arguments of tool calls and the contents of tool call responses. The arguments of tool calls are shared between choice and message events. However both of these have type undefined and are likely to not fit standard attribute types.