How to transport multimodel messages like images, files?

[NiuBlibing]

Currently, AG-UI only supports text-based messages. However, in recent practical agent applications, the ability to handle multimodal input/output messages—even general files—has become increasingly critical. Imagine an agent designed for creating presentations: users input text, images, and video materials, and the agent retrieves relevant information to generate a complete PowerPoint file. While using custom extensions can temporarily support such functionality, it introduces fragmentation at the protocol level.

I propose establishing certain standardized specifications at the protocol layer to unify support and avoid confusion.

Supported Content:

User Input:

• Text
• Audio
• Images
• General Files

Agent Output (AssistantMessage, UserMessage, ToolMessage):

• Text
• Audio
• Images
• General Files

Similar issues:
#26 #77 #280

[widike]

That would be amazing. The Copilotkit Chat allows to send images but they get lost in AG-UI.

[NiuBlibing]

The following are the messages format of openai, mcp and A2A:

openai

text

{"type": "input_text", "text": "what is in this image?"}

image

{"type": "input_image","image_url": "<url> or <base64 bytes>"}

audio

https://platform.openai.com/docs/guides/audio?example=audio-in

{ "type": "input_audio", "input_audio": {   "data": "<base64 bytes here>",   "format": "wav" }}

file

https://platform.openai.com/docs/guides/pdf-files?api-mode=chat

{
    "type": "file",
    "file": {
        "file_id": "file-6F2ksmvXxt4VdoqmHRw6kL"
    }
}

{
    "type": "file",
    "file": {
        "filename": "draconomicon.pdf",
        "file_data": "...base64 encoded bytes here..."
    }
}

mcp

mcp supports TextContent, ImageContent, AudioContent
https://modelcontextprotocol.io/specification/2025-06-18/schema

text

/**
 * Text provided to or from an LLM.
 */
export interface TextContent {
  type: "text";

  /**
   * The text content of the message.
   */
  text: string;

  /**
   * Optional annotations for the client.
   */
  annotations?: Annotations;

  /**
   * See [General fields: `_meta`](/specification/draft/basic/index#meta) for notes on `_meta` usage.
   */
  _meta?: { [key: string]: unknown };
}

image

export interface ImageContent {
  type: "image";

  /**
   * The base64-encoded image data.
   *
   * @format byte
   */
  data: string;

  /**
   * The MIME type of the image. Different providers may support different image types.
   */
  mimeType: string;

  /**
   * Optional annotations for the client.
   */
  annotations?: Annotations;

  /**
   * See [General fields: `_meta`](/specification/draft/basic/index#meta) for notes on `_meta` usage.
   */
  _meta?: { [key: string]: unknown };
}

audio

export interface AudioContent {
  type: "audio";

  /**
   * The base64-encoded audio data.
   *
   * @format byte
   */
  data: string;

  /**
   * The MIME type of the audio. Different providers may support different audio types.
   */
  mimeType: string;

  /**
   * Optional annotations for the client.
   */
  annotations?: Annotations;

  /**
   * See [General fields: `_meta`](/specification/draft/basic/index#meta) for notes on `_meta` usage.
   */
  _meta?: { [key: string]: unknown };
}

file

not support

A2A

A2A uses part messages like openai parts. https://a2a-protocol.org/latest/specification/#65-part-union-type

text

/**
 * Represents a text segment within a message or artifact.
 */
export interface TextPart extends PartBase {
  /** The type of this part, used as a discriminator. Always 'text'. */
  readonly kind: "text";
  /** The string content of the text part. */
  text: string;
}

audio

image

file

/**
 * Defines base properties for a file.
 */
export interface FileBase {
  /** An optional name for the file (e.g., "document.pdf"). */
  name?: string;
  /** The MIME type of the file (e.g., "application/pdf"). */
  mimeType?: string;
}

/**
 * Represents a file with its content provided directly as a base64-encoded string.
 */
export interface FileWithBytes extends FileBase {
  /** The base64-encoded content of the file. */
  bytes: string;
  /** The `uri` property must be absent when `bytes` is present. */
  uri?: never;
}

/**
 * Represents a file with its content located at a specific URI.
 */
export interface FileWithUri extends FileBase {
  /** A URL pointing to the file's content. */
  uri: string;
  /** The `bytes` property must be absent when `uri` is present. */
  bytes?: never;
}

/**
 * Represents a file segment within a message or artifact. The file content can be
 * provided either directly as bytes or as a URI.
 */
export interface FilePart extends PartBase {
  /** The type of this part, used as a discriminator. Always 'file'. */
  readonly kind: "file";
  /** The file content, represented as either a URI or as base64-encoded bytes. */
  file: FileWithBytes | FileWithUri;
}

data

/**
 * Represents a structured data segment (e.g., JSON) within a message or artifact.
 */
export interface DataPart extends PartBase {
  /** The type of this part, used as a discriminator. Always 'data'. */
  readonly kind: "data";
  /** The structured data content. */
  data: {
    [key: string]: any;
  };
}

[NiuBlibing]

Hi, @NathanTarbert @mme Is there any progress? I propose a draft pr #196 for this issue, if the modify of the protocol is ok, I will continue the typescirpt part.

[mme]

@NiuBlibing We made this draft proposal how to change AG-UI to support multi-modal input. Curious about your feedback.

https://docs.ag-ui.com/drafts/multimodal-messages

[anubhabPanda]

Any update on this? Is this merged @NiuBlibing @mme

[NiuBlibing]

@NiuBlibing We made this draft proposal how to change AG-UI to support multi-modal input. Curious about your feedback.

https://docs.ag-ui.com/drafts/multimodal-messages

@mme Great! It fit most usages. And AssistantMessage also need string | InputContent[], so we can generate images, video, files from agents to users.