GenAI handle tool message embedded within user message

[aabmass]

Area(s)

area:gen-ai

What's missing?

OpenAI treats tool call responses as a separate "role" with its own type. VertexAI instead represents tool responses as another variant of Part which can appear without a role or within a user role (function calling docs), for example

{
  "role": "user",
  "parts": [
    {
      "functionResponse": {
        "name": "get_current_weather",
        "response": {
          "content": "{\"temperature\": 35, \"unit\": \"C\"}"
        }
      }
    },
    {
      "functionResponse": {
        "name": "get_current_weather",
        "response": {
          "content": "{\"temperature\": 25, \"unit\": \"C\"}"
        }
      }
    },
    {
      "text": "Please respond in French"
    }
  ]
}

(Full example in open-telemetry/opentelemetry-python-contrib@f322c46)

This doesn't really fit with the semconv for gen_ai.tool.message. Vertex has overlap of user event and tool events, similar to how tool call requests are embedded in gen_ai.choice events.

Describe the solution you'd like

We could either adapt the semconv to embed tool calls in gen_ai.user.message event or keep the current semconv and massage Vertex response into it.

Possible options for massaging the data:

1. One gen_ai.user.message with just "Please respond in French" content plus two gen_ai.tool.message for the tool calls.
2. One gen_ai.user.message event with all three parts in it plus two gen_ai.tool.message duplicating the tool calls.

The benefit of 2. is for debugging you can actually reconstruct the exact original API request.

[aabmass]

There is a similar issue for tool call requests from returned from the vertex ai model.

Since function_call appears in content.parts, it is duplicated in the gen_ai.choice event's message and tool_calls body fields. This is different from OpenAI where the tool calls are outside of content: https://platform.openai.com/docs/api-reference/chat/object. I would prefer not to filter it out of message so that it is easy and obvious how to reconstruct the original API call.

Example Log body

{
  "message": {
    "role": "model",
    "content": [
      {
        "text": "It seems like SQLite doesn't support adding a primary key column after the table has been created. Let's try a different approach. We can create the table again with the ID column included. \n\n"
      },
      {
        "function_call": {
          "name": "sql_db_query",
          "args": {
            "query": "CREATE TABLE weather_data (id INTEGER PRIMARY KEY AUTOINCREMENT, date DATE, location VARCHAR(255), temperature FLOAT, humidity FLOAT, wind_speed FLOAT, conditions VARCHAR(255))"
          }
        }
      }
    ]
  },
  "tool_calls": [
    {
      "type": "function",
      "function": {
        "name": "sql_db_query",
        "arguments": {
          "query": "CREATE TABLE weather_data (id INTEGER PRIMARY KEY AUTOINCREMENT, date DATE, location VARCHAR(255), temperature FLOAT, humidity FLOAT, wind_speed FLOAT, conditions VARCHAR(255))"
        }
      },
      "id": "sql_db_query_1"
    }
  ],
  "index": 0,
  "finish_reason": "stop"
}

[alexmojaki]

Proposal for events that contain both the text and tool calls as separate parts in a single message and thus look more like the API request: #1913

[xrmx]

I don't have a strong opinion on this but aren't we already making up the response in case of streaming responses?