fix(google): strip media from functionResponse, preserve as sibling parts by pawel-twardziak · Pull Request #10417 · langchain-ai/langchainjs

pawel-twardziak · 2026-03-14T15:47:50Z

When a ToolMessage contains multimodal content (text + images via image_url blocks), the converter serializes inlineData/fileData inside functionResponse.response.result as JSON text. Gemini cannot interpret images nested inside a JSON string - they become invisible to the model.

This PR fixes both the legacy and v1 converter paths to:

Exclude media items (image_url, media, image, audio, video, file) from functionResponse.response.result, keeping it text-only
Preserve media as sibling parts (inlineData/fileData) alongside functionResponse, where Gemini can interpret them natively
Filter out redundant text parts from tool responses - their content is already in functionResponse.response.result

Also adds @langchain/google to the CI unit test matrix - it was missing from the PACKAGES list in unit-tests-integrations.yml, so its tests were never run in CI.

#10415 - this PR must be merged first - it has some tests fixed that previously couldn't even get run in CI/CD - UPDATE: this is not valid any longer

Fixes #10297

…arts

changeset-bot · 2026-03-14T15:47:55Z

🦋 Changeset detected

Latest commit: 29ebf67

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 6 packages

Name	Type
@langchain/google-common	Patch
@langchain/google-gauth	Patch
@langchain/google-vertexai-web	Patch
@langchain/google-vertexai	Patch
@langchain/google-webauth	Patch
@langchain/google-genai	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Allen Firstenberg (afirstenberg) · 2026-03-14T17:21:20Z

pawel-twardziak - Thank you for this!

This addresses approach 2 from #10297, right?
Do we know if this works on Gemini 2.5?

And do we have an integration test case that illustrates the issue?

pawel-twardziak · 2026-03-14T19:59:11Z

hi, thanks for reviewing, Allen Firstenberg (@afirstenberg) !

as far as I understood - this actually implements approach 1 (sibling parts) from #10297 - inlineData/fileData are extracted as separate Part objects alongside functionResponse. This is the approach Antonio Pinto (@Toze) confirmed works across model versions in their local patch.

Regarding Gemini 2.5: yes, the sibling parts approach works on Gemini 2.5 (and 3). Since Content.parts is repeated Part, placing functionResponse and inlineData in separate Part objects is structurally valid per the protobuf definition.

For approach 2 (FunctionResponse.parts for Gemini 3+), that could be a follow-up PR - per your earlier suggestion of supporting "both" with a model version check. Would you like me to tackle that as well?

Regarding integration tests - yes, we can add one. The existing suite in index.int.test.ts already has both tool conversation tests (e.g. "function conversation") and image tests (using blue-square.png). A test for this fix would combine them:

Define a tool (e.g. screenshot_tool)
Build a message history with a ToolMessage containing [{type: "text", text: "screenshot"}, {type: "image_url", image_url: {url: "data:image/png;base64,..."}}] using the existing blue-square.png
Invoke the model and assert the response mentions "blue" - proving the model visually interpreted the image rather than seeing serialized base64 text

pawel-twardziak · 2026-03-14T20:09:05Z

the branch name is wrong 🤷‍♂️

Allen Firstenberg (afirstenberg) · 2026-03-14T21:00:28Z

pawel-twardziak

as far as I understood - this actually implements approach 1

Ah, ok. I hadn't had a chance to look closely at everything yet.

For approach 2 (FunctionResponse.parts for Gemini 3+), that could be a follow-up PR - per your earlier suggestion of supporting "both" with a model version check. Would you like me to tackle that as well?

That would be fantastic - thank you! A model check plus a way to override would be great.

I think you're right that doing it as a second PR would be best. Let's get this in there, but plan to have an update coming.

(This is also why I'd like to see an integration test.)

Regarding integration tests - yes, we can add one.

Very much appreciated. I didn't think it would be hard to do, but wanted to make sure I understood the problem before we fixed it. Also helps to make sure we dont' break things in the future.

If you can add the test, I can go run it in the pre-patch environment to give it a try.

Thanks again!

pawel-twardziak · 2026-03-14T21:16:31Z

Alright, thanks for your input Allen Firstenberg (@afirstenberg)

the plan is:

within this PR I'll add some integration tests
the approach 2 will be addressed in the next follow-up pr

pawel-twardziak · 2026-03-15T16:07:22Z

hi Allen Firstenberg (@afirstenberg) I just sent over the tests - the PR should be ready for your review

pawel-twardziak · 2026-03-15T23:43:53Z

Blocked by #10415 - UPDATE: not valid any longer

Allen Firstenberg (afirstenberg) · 2026-03-16T02:15:11Z

I'm confused why this is blocked by #10415. Just for the tests? (Can we just move the tests to this one?)

Allen Firstenberg (afirstenberg)

I need to look more closely at this, but here are some initial thoughts and comments.

Allen Firstenberg (afirstenberg) · 2026-03-16T02:17:44Z

libs/providers/langchain-google/src/chat_models/tests/index.int.test.ts

    });

    test("function reply", async () => {
+      // Thinking models require thought_signature on functionCall parts,


I'm not sure I understand this addition. Have we seen issues along these lines with this test? (I haven't before, I don't think.) And is this not a "real model invocation"?

Yes, this issue:

RequestError: Function call is missing a thought_signature in functionCall parts. This is required for tools to work correctly, and missing thought_signature may lead to degraded model performance. Additional data, function call default_api:test , position 2. Please refer to https://ai.google.dev/gemini-api/docs/thought-signatures for more details.

This is a real model invocation

Allen Firstenberg (afirstenberg) · 2026-03-16T02:19:56Z

libs/providers/langchain-google/src/chat_models/tests/index.int.test.ts

      if (model.startsWith("gemini-2.0-flash")) {
        return;
      }
+      // Not available on Gemini 3 models


There is backwards compatibility which maps it to the Google Search tool. So while Google doesn't support it - we do.
If this test fails - then something is broken with the backwards compatibility.

none of the tests fail - checked on local

yes, it fails when I remove it:

AssertionError: expected { tokenUsage: { …(5) }, …(11) } to have property "url_context_metadata"

My mistake - I thought this was the search tool, not URL Context tool.
And... this used to pass.
https://ai.google.dev/gemini-api/docs/url-context says it is supported.

If this is failing - something else is going on.

do the limitations have something to do with it https://ai.google.dev/gemini-api/docs/url-context?hl=en#limitations ?

gemini-3-flash-preview IS in the supported models list. So it should work but doesn't return url_context_metadata.

That's likely a preview model quirk - it's documented as supported but the response doesn't include the metadata consistently. I added a comment and a warning about it.

libs/providers/langchain-google/src/chat_models/tests/index.int.test.ts

Allen Firstenberg (afirstenberg) · 2026-03-16T02:25:18Z

libs/providers/langchain-google/src/chat_models/tests/index.int.test.ts


      expect(result.text as string).toMatch(/(1 + 1 (equals|is|=) )?2.? ?/);
-      // With includeThoughts: true, response may have multiple parts (reasoning + text)
+      // Thought signatures are best-effort per Google docs; only assert when


Citation? (It seems like we run the risk of a false failure if we only test for thought signatures when we have thought signatures.)

will check it again

I'm not sure I follow this comment. Could you rephrase it please?

Allen Firstenberg (afirstenberg) · 2026-03-16T02:26:28Z

libs/providers/langchain-google/src/converters/messages.ts

-    role = "function";
+    // Tool messages in Gemini are represented as function responses.
+    // Recent Gemini API versions require 'user' or 'model' role for all contents.
+    role = "user";


ok, will do

libs/providers/langchain-google/src/converters/messages.ts

Allen Firstenberg (afirstenberg) · 2026-03-16T02:42:33Z

libs/providers/langchain-google/src/converters/messages.ts

    );
    parts.push({
      functionResponse: {
-        id: message.tool_call_id,


There are cases where there is an ID that needs to be returned. This should not just be removed.
See #10292

dozens of integration tests fail if I don't remove it. Will check it again

RequestError: Invalid JSON payload received. Unknown name "id" at 'contents[2].parts[1].function_response': Cannot find field.

there is no id for Vertex AI https://docs.cloud.google.com/vertex-ai/generative-ai/docs/reference/rest/v1/Content#FunctionResponse

and for Google AI it's optional and kinda useless https://ai.google.dev/api/caching?authuser=1&hl=pl#FunctionResponse

we have GooglePlatformType = "gai" | "gcp" so we could conditionally include id. But to do so, we'd need to thread platformType through the entire converter call chain - a fairly invasive change for a field that:

doesn't exist on Vertex AI at all

gets rejected by Gemini 3 preview models even on Google AI

The pragmatic call for now: just remove id. It's not usable on either platform in practice.

We had use cases where ID was sent and, since it wasn't returned, it was rejected. I, honestly, have no idea what the situations were.

This is handled in #10292.

so then we might need the conditional way of handling it

Which #10292 does.

imho we should skip it for now since all tests are good without this id param - unless we know the exact case when it's required

We do. It's in #10292 🙂

pawel-twardziak · 2026-03-16T07:52:56Z

Allen Firstenberg (@afirstenberg) thanks for you detailed review! Really appreciated 💪
fyi, all the integration tests pass in this PR's current state - I've checked it on local a few times

pawel-twardziak · 2026-03-16T08:05:26Z

I'm confused why this is blocked by #10415. Just for the tests? (Can we just move the tests to this one?)

that PR was first fixing the "function -> user" change - but I don't see that change there now, strange... I must have mixed something up. Will add the change here in this PR

…ring

fix(google): strip media from functionResponse, preserve as sibling p…

a889a25

…arts

github-actions bot added the ci label Mar 14, 2026

fix(google): update ToolMessage converter for latest Gemini API

7e7922d

pawel-twardziak added 2 commits March 15, 2026 18:24

chore: add patch for google-common ToolMessage fixes

280f1f7

Merge branch 'main' into fix/langchain-google-missing-ci-tests

1332502

chore: add test:int script to root package.json

ee36c44

pawel-twardziak force-pushed the fix/langchain-google-missing-ci-tests branch from 956a8b3 to ee36c44 Compare March 15, 2026 23:50

fix(google): type-safe thoughtSignature and optional parts access

82f8c93

Allen Firstenberg (afirstenberg) suggested changes Mar 16, 2026

View reviewed changes

pawel-twardziak added 4 commits March 16, 2026 09:13

test(google): fix ToolMessage role assertions in converter tests

5b266dd

fix(google): address the PR review feedback

a51c890

Merge branch 'main' into fix/langchain-google-missing-ci-tests

a5748ce

fix(google): correct ToolMessage role, id passthrough, and part filte…

29ebf67

…ring

Conversation

pawel-twardziak commented Mar 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot bot commented Mar 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

Allen Firstenberg (afirstenberg) commented Mar 14, 2026

Uh oh!

pawel-twardziak commented Mar 14, 2026

Uh oh!

pawel-twardziak commented Mar 14, 2026

Uh oh!

Allen Firstenberg (afirstenberg) commented Mar 14, 2026

Uh oh!

pawel-twardziak commented Mar 14, 2026

Uh oh!

pawel-twardziak commented Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pawel-twardziak commented Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Allen Firstenberg (afirstenberg) commented Mar 16, 2026

Uh oh!

Allen Firstenberg (afirstenberg) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pawel-twardziak Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pawel-twardziak Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pawel-twardziak Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

pawel-twardziak commented Mar 14, 2026 •

edited

Loading

changeset-bot bot commented Mar 14, 2026 •

edited

Loading

pawel-twardziak commented Mar 15, 2026 •

edited

Loading

pawel-twardziak commented Mar 15, 2026 •

edited

Loading

pawel-twardziak Mar 16, 2026 •

edited

Loading

pawel-twardziak Mar 16, 2026 •

edited

Loading

pawel-twardziak Mar 16, 2026 •

edited

Loading