Images in ToolMessage content lost when converted to Gemini functionResponse

[Toze]

Bug Report

Description

When a ToolMessage contains multimodal content (text + images via image_url blocks), the @langchain/google converter stuffs the inlineData (base64 images) inside the functionResponse.response.result as serialized JSON. Gemini's functionResponse only accepts JSON — it cannot interpret inlineData nested inside it. The images are effectively invisible to the model.

Environment

• @langchain/google: 0.1.5
• @langchain/core: 1.1.31
• Model: gemini-3-flash-preview (also affects gemini-2.5-flash)

Reproduction

import { ToolMessage, AIMessage, HumanMessage } from '@langchain/core/messages'

const messages = [
  new HumanMessage('What is in the attachment?'),
  new AIMessage({
    content: '',
    tool_calls: [{ id: 'call_123', name: 'read_file', args: { path: 'image.png' } }]
  }),
  new ToolMessage({
    content: [
      { type: 'text', text: 'Image: screenshot.png (133KB)' },
      {
        type: 'image_url',
        image_url: { url: 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUg==' }
      }
    ],
    tool_call_id: 'call_123',
    name: 'read_file'
  })
]

Expected Behavior

The converted Gemini content should have inlineData as a sibling part alongside functionResponse, so Gemini can visually interpret the image:

{
  "role": "function",
  "parts": [
    {
      "functionResponse": {
        "name": "read_file",
        "response": { "result": "Image: screenshot.png (133KB)" }
      }
    },
    {
      "inlineData": {
        "mimeType": "image/png",
        "data": "iVBORw0KGgoAAAANSUhEUg=="
      }
    }
  ]
}

Actual Behavior

inlineData is serialized as JSON inside functionResponse.response.result:

{
  "role": "function",
  "parts": [
    {
      "functionResponse": {
        "name": "read_file",
        "response": {
          "result": "[{\"type\":\"text\",\"text\":\"Image: screenshot.png (133KB)\"},{\"inlineData\":{\"mimeType\":\"image/png\",\"data\":\"iVBORw0KGgoAAAANSUhEUg==\"}}]"
        }
      }
    }
  ]
}

The model receives the base64 string as text, not as visual data. It either hallucinates the image content or fails to interpret it.

Root Cause

In src/converters/messages.ts, both convertStandardContentMessageToGeminiContent and convertLegacyContentMessageToGeminiContent convert all ToolMessage content parts into a single functionResponse. When the content is multimodal (contains image_url blocks), the inlineData parts get serialized into the response.result JSON string rather than being extracted as sibling Gemini content parts.

Suggested Fix

When processing a ToolMessage, extract inlineData and fileData parts before building the functionResponse, then add them as sibling parts:

if (ToolMessage.isInstance(message) && message.tool_call_id) {
  const mediaParts = parts.filter((p) => "inlineData" in p || "fileData" in p);
  const textParts = parts.filter((p) => !("inlineData" in p) && !("fileData" in p));
  
  const responseContent = typeof message.content === "string"
    ? message.content
    : textParts.map((p) => p.text || "").filter(Boolean).join("\n") || JSON.stringify(message.content);
  
  parts.length = 0;
  parts.push(
    {
      functionResponse: {
        id: message.tool_call_id,
        name: message.name || "unknown",
        response: { result: responseContent }
      }
    },
    ...mediaParts
  );
}

This keeps functionResponse as pure JSON (text only) and places inlineData/fileData as sibling parts where Gemini can interpret them visually.

Impact

Any agent that returns images from tool calls (e.g., reading files, fetching screenshots, downloading attachments) will have the images silently dropped when using Gemini models through @langchain/google. The model receives only the base64 string as text and either hallucinates the image description or fails.

Workaround

We are currently using a pnpm patch on @langchain/google to extract media parts from ToolMessage content before building the functionResponse.

[afirstenberg]

Antonio Pinto (@Toze) - Thanks for this report!

Is the expected behavior documented somewhere from Google as valid. I agree - what its doing sounds wrong, but I want to make sure we're following a documented correct approach somewhere.

It sounds like you may already have a fix? If so, feel free to submit it. Otherwise, Hunter Lovell (@hntrl) - please assign to me.

[Toze]

Allen Firstenberg (@afirstenberg) Thanks for the quick reply!

Documentation

The Gemini API's Part message uses a oneof data field in the protobuf definition (content.proto):

message Part {
  oneof data {
    string text = 2;
    Blob inline_data = 3;
    FunctionCall function_call = 4;
    FunctionResponse function_response = 5;
    FileData file_data = 6;
    ExecutableCode executable_code = 9;
    CodeExecutionResult code_execution_result = 10;
  }
}

The Vertex AI REST reference confirms: Content.Part.data is a union field.

Since Content.parts is repeated Part, putting functionResponse and inlineData in separate Part objects within the same array is structurally valid.

Additionally, for Gemini 3+ models, Google introduced a dedicated mechanism: FunctionResponse.parts with FunctionResponsePart / FunctionResponseBlob. Documented under "Multi-modal function responses" (search for "multimodal function responses"). The @langchain/google types already have FunctionResponse.parts, FunctionResponsePart, and FunctionResponseBlob defined.

Two possible approaches for the fix

1. Sibling parts — extract inlineData/fileData from the converted parts and place them as separate Part objects alongside the functionResponse Part. This is what our local patch does and works across all model versions.

2. FunctionResponse.parts (official) — nest the media inside functionResponse.parts as FunctionResponsePart objects. This is the Google-documented approach but is Gemini 3+ only.

Happy to submit a PR for either approach. Do you have a preference?

[Toze]

Update: additional bug — functionResponse.id rejected by Gemini 3

While testing with Gemini 3 Flash, we hit a second issue in the same converter: functionResponse includes an id field (id: message.tool_call_id) that is not part of the Gemini API's FunctionResponse schema. Gemini 2.5 silently ignored unknown fields, but Gemini 3 is strict:

RequestError: Invalid JSON payload received. Unknown name "id" at 'contents[14].parts[0].function_response': Cannot find field.

The protobuf definition for FunctionResponse only has name and response — no id field. We've added this removal to our local patch as well.

Note: the suggested fix in the issue description above also includes id: message.tool_call_id — that should be removed too.

We also filed this as a separate issue: #10306 (error body parsing), but the id field issue is specific to this converter code.

[afirstenberg]

While testing with Gemini 3 Flash, we hit a second issue in the same converter: functionResponse includes an id field (id: message.tool_call_id) that is not part of the Gemini API's FunctionResponse schema.

This is addressed in #10292

[afirstenberg]

Antonio Pinto (@Toze) - Thanks for the reference, that's exactly what I was looking for.

My inclination is to go with "both" and have it use sibling parts for Gemini 2.5 and the new documented way otherwise. I'm cautious about relying on a non-documented method going forward because, as we've seen, they change things somewhat inconsistently. But I'd like to see the compatibility. This is consistent with approaches we've taken on some other things (such as Google Search support changing between 2.0 and 2.5).

Thanks again!