Troubleshooting image content from MCP tool's return

[ElgoogUdiab]

What features would you like to see added?

Currently librechat cannot correctly decode image content returned from the MCP tools but only as an extra long base64 strings. I hope that Image Content can be correctly decoded and add to the context if receiving images from MCP tools.

More details

I'm pretty sure that the tool is returning the image in the right way since Claude Desktop can correctly decode the image. And I'm sure that I'm using a model having vision capabilities since I can manually upload the image to the model.

Which components are impacted by your request?

General

Pictures

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

When I updated the MCP SDK, it seems image content had changed and I was not aware of it. The intention is not just to display base64 as the image representation and to actually show the image

[danny-avila]

Actually, it's already working as expected with OpenAI and custom endpoints

there is an issue with OpenRouter, since they don't support returning images from tool outputs

[danny-avila]

Anthropic direct also has no issue

[ElgoogUdiab]

I'm using the version built from commit 150116e and using my custom MCP tool via SSE. My server returns the image using fastmcp's Image class. My custom endpoint raised an error like this:

Something went wrong. Here's the specific error message we encountered: An error occurred while processing thee request:
400 litellm.ContextWindowExceedededError: litellm.BadRegjuestError: ContextWindowExceededError: OpenAlExcepton - This
model's maximum context length is 300000 tokens. However, your messages resulted in 321338 tokens (318735 in the
messages, 2603 in the functions). 

The tool's output is a raw json block having type, image, and MIME like this:

{
  "type": "image",
  "data": "iVBORw0KGgoAAAANSUhEUgAAAwgAAAMIC..."
  "mimeType": "image/png"
}

I've pasted the tool's output to OpenAI's tokenizer, showing the output has 317505 tokens.

I've tested the tool on Claude Desktop, and it can correctly decode the image.

[danny-avila]

I'm making some improvements here soon, but what exactly is your custom endpoint config? In any case, I'm improving it for xAI, OpenRouter, etc.

[ElgoogUdiab]

I'm using Claude 3.7 Sonnet (via google vertex ai) and GPT-4.1 (via official API endpoint) through a self-deployed litellm.

[danny-avila]

Closing this: #7040

[wrc92]

I'm seeing a similar error with gemini-2.5-pro-preview-05-06 on Vertex AI. Although the base64 encoded image is not filling up the context window, the model says that the image is being shown as a base64 string.