Support for Multi modal inputs and generations

[karthikscale3]

Area(s)

area:gen-ai

Is your change request related to a problem? Please describe.

This relates to the support for multi modal content - images/audio etc. (inputs and generation). The current GenAI spec does not have support for multi modal inputs and generations as there needs to be additional considerations around the increase the size of the span. As a result, an external blob storage is being considered for storage and hyperlinking the stored location back to the span.

The main task relates to:

• Replace content attribute in event with a similar structure to OpenAI - array of content

Some additional comments and open questions discussed during the SIG meeting:

• We should add a hook in the instrumentation (simiar to OpenLLMetry instrumentation)
• Semantic convention for references - Add the concept of Blob Reference Properties ("reference attributes" and "reference fields") to the spec #1521
• What metadata should we stored with the blob that is stored in the bucket?

Describe the solution you'd like

The main task relates to:

• Replace content attribute in event with a similar structure to OpenAI - array of content

Describe alternatives you've considered

No response

Additional context

No response

[lmolkova]

Related to #963

[alexmojaki]

Superset of this proposal: #1913

[alexmojaki]

Examples from provider docs

https://platform.openai.com/docs/guides/pdf-files?api-mode=responses

{
    "type": "input_file",
    "file_url": "https://www.berkshirehathaway.com/letters/2024ltr.pdf",
}
{
    "type": "input_file",
    "file_id": file.id,
}
{
    "type": "input_file",
    "filename": "draconomicon.pdf",
    "file_data": f"data:application/pdf;base64,{base64_string}",
}

https://platform.openai.com/docs/guides/images-vision

{
    "type": "input_image",
    "image_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
}
{
    "type": "input_image",
    "file_id": file_id,
}
{
    "type": "input_image",
    "image_url": f"data:image/jpeg;base64,{base64_image}",
}

https://platform.openai.com/docs/guides/audio?example=audio-in

{
    "type": "input_audio",
    "input_audio": {
        "data": base64.b64encode(wav_data).decode('utf-8'),
        "format": "wav"
    }
}

https://ai.google.dev/gemini-api/docs/image-understanding

  types.Part.from_bytes(
    data=image_bytes,
    mime_type='image/jpeg',
  ),

  const myfile = await ai.files.upload({
    file: "path/to/sample.jpg",
    config: { mimeType: "image/jpeg" },
  });

  const response = await ai.models.generateContent({
    model: "gemini-2.5-flash",
    contents: createUserContent([
      createPartFromUri(myfile.uri, myfile.mimeType),

Info to record:

• The general type of a file: image, audio, video, document
• File format, e.g. jpeg or wav. Could be combined with the above into a single mimetype field.
• The data itself: either inline base64, or a URL, or an identifier of a previously uploaded file
• Other metadata, generally vendor-specific, e.g. the filename for inline data in OpenAI, or the sampling frame rate for videos in Gemini.

Questions:

• Do we have a separate top level part type for each kind of file/media, or do we have a single type (e.g. media) and a subfield to differentiate? They all seem very similar.
• Do we separate mimetype into two fields?
• Do we distinguish between inline data and URLs? Note that e.g. OpenAI's image_url field doesn't, it just uses a data: URL for inline data. This seems pretty simple and convenient. Maybe an additional boolean to indicate which it is? Or people could just check if the URL starts with data:. Alternatively, we have two different types, e.g. media_inline and media_url, or 2 per media type e.g. image_inline etc.
• Should the OpenAI file_id be considered something vendor-specific that doesn't really need a unified semantic convention? It's not like telemetry consumers can do much with it, in contrast to the other data which allows rendering in a UI.

[alexmojaki]

Example solution:

{
  type: 'media',
  media_type: 'image',
  format: 'jpeg',
  url: 'data:image/jpeg;base64,{base64_image}',
}