fix(core): binary content handling in parallel function responses

[jw409]

Fixes #17068

[gemini-code-assist]

Summary of Changes

Hello @jw409, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the handling of binary content, specifically fileData and inlineData, within parallel function responses to prevent API 400 errors caused by an incorrect part count. It introduces a conditional logic that, for Gemini 3+ models supporting multimodal function responses, nests binary content directly within the functionResponse.parts structure. For older Gemini 2.x models that do not support this, binary content is now explicitly omitted from the response, accompanied by a debug warning, ensuring API compatibility while acknowledging the limitation. The core aim is to maintain a strict 1:1 mapping between function calls and their responses, crucial for parallel execution.

Highlights

• Problem Addressed: Fixed API 400 errors occurring when binary content (fileData, inlineData) was returned in parallel function responses due to mismatched part counts.
• Gemini 3+ Handling: Binary content is now nested directly within functionResponse.parts to maintain a 1:1 mapping between function calls and responses, enabling multimodal function responses.
• Gemini 2.x Handling: For models without multimodal support, sibling binary parts are now omitted from the response, with a debug warning logged, to prevent API errors.
• Refactoring: The changes were extracted from a larger pull request (fix: Comprehensive fix for parallel function calling 400 errors #16146) to focus solely on binary content handling, as per reviewer feedback.
• Test Coverage: Updated unit tests confirm the new nesting behavior for Gemini 3+ and the omission of binary parts for Gemini 2.x models.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request correctly refactors the handling of binary content in function responses to support parallel function calls. The core change properly nests fileData and inlineData for multimodal-capable models (like Gemini 3) and drops them for older models to prevent API errors, which is a solid improvement.

I've identified one high-severity issue where a misleading descriptive text ('Binary content provided...') is added to the response for older models, even though the binary content itself is dropped. This could confuse the model. My review includes a specific suggestion to fix this by ensuring the text is only added when the binary content is actually sent.

Overall, the changes are well-structured and address the stated problem effectively, pending the fix for the identified issue.

[gemini-code-assist]

This logic correctly warns and drops sibling parts for non-multimodal models. However, there's a related issue: the code at lines 115-123 has already added a descriptive text like output: 'Binary content provided (...)' to the response. This message becomes misleading because the binary content is now being dropped. The model will be told binary content is provided when it has actually been omitted.

To fix this, we should clear this misleading message from the response object when we drop the parts and there was no original text content. This will also require updating the test 'should handle llmContent with fileData for non-Gemini 3 models (should omit siblings)' to expect response: {}.

Suggested change

	if (siblingParts.length > 0) {
	return [part, ...siblingParts];
	debugLogger.warn(
	`Model ${model} does not support multimodal function responses. Sibling parts will be omitted to prevent API errors in parallel function calling.`,
	);
	}
	if (siblingParts.length > 0) {
	debugLogger.warn(
	`Model ${model} does not support multimodal function responses. Sibling parts will be omitted to prevent API errors in parallel function calling.`,
	);
	// If there was no original text, the response object may contain a misleading
	// "Binary content provided" message from the dropped parts. Clear it.
	if (textParts.length === 0) {
	part.functionResponse!.response = {};
	}
	}

[jw409]

Addressed in 0e4cf4e:

• For multimodal models (Gemini 3+): "Binary content provided (N item(s))." - accurate, binary is nested
• For non-multimodal models (Gemini 2.x): "Tool executed but binary output was omitted (model does not support multimodal function responses)." - accurate, binary is dropped

This prevents misleading the model about what content is available.

[jw409]

@gemini-code-assist

[gemini-code-assist]

Thanks for confirming, @jw409! It's great to hear that the binary content handling is now accurate for both Gemini 3+ (nested) and Gemini 2.x (omitted with warning) models. This ensures correct API behavior and clear communication about content availability. The solution looks robust.

[jw409]

Fixes #17068

[gemini-cli]

Hi there! Thank you for your contribution to Gemini CLI. We really appreciate the time and effort you've put into this pull request.

To keep our backlog manageable and ensure we're focusing on current priorities, we are closing pull requests that haven't seen maintainer activity for 30 days. Currently, the team is prioritizing work associated with 🔒 maintainer only or help wanted issues.

If you believe this change is still critical, please feel free to comment with updated details. Otherwise, we encourage contributors to focus on open issues labeled as help wanted. Thank you for your understanding!