Bedrock/Azure models do not handle properly image returned by MCP tool

[dvejsada]

What happened?

When using Amazon Bedrock Anthropic models or Ms Azure OpenAi models in agent with MCP tool that returns image as base64 (from Amazon Nova Canvas model), agent provides an error:

2025-01-27 20:25:59 error: [api/server/controllers/agents/client.js #sendCompletion] Unhandled error type 400 This model's maximum context length is 128000 tokens. However, your messages resulted in 375014 tokens (374020 in the messages, 994 in the functions). Please reduce the length of the messages or functions.
2025-01-27 20:25:59 error: [handleAbortError] AI response error; aborting request: 400 This model's maximum context length is 128000 tokens. However, your messages resulted in 375014 tokens (374020 in the messages, 994 in the functions). Please reduce the length of the messages or functions.

The tool image output works fine with OpenAI and Anthropic (not via Bedrock/MsAzure)

Steps to Reproduce

1. Have agent with MCP tool providing images with Bedrock/Azure model
2. Try to generate image
3. See error

What browsers are you seeing the problem on?

Chrome

Relevant log output

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

It's a provider limitation. Even using latest Claude on Bedrock, you'll get this when adding an image to content blocks:

ValidationException: The format of the value at messages.2.content.0.toolResult.content.0.json is invalid. Provide a json object for the field and try again.

Even though this is accepted by Anthropic API.

As for Azure, it should work, since I just format the payload similar to a vision request (and OpenAI API allows many different combinations of role order in messages):

  {
    type: 'text',
    text: '',
    tool_call_ids: [ 'call_suftWTQ3GDcsBd7yfeGrOAuM' ]
  },
  {
    type: 'tool_call',
    tool_call: {
      id: 'call_suftWTQ3GDcsBd7yfeGrOAuM',
      name: 'fetchRandomImage',
      args: '{}',
      type: 'tool_call',
      progress: 1,
      output: '[{"type":"text","text":"Random image from Lorem Picsum, taken at 800x600"}]'
    }
  },
  {
    type: 'text',
    text: '',
    tool_call_ids: [ 'call_YfnuE8zViHs4tEVxXpHmlzJ1' ]
  },
  {
    type: 'tool_call',
    tool_call: {
      id: 'call_YfnuE8zViHs4tEVxXpHmlzJ1',
      name: 'fetchRandomImage',
      args: '{}',
      type: 'tool_call',
      progress: 1,
      output: '[{"type":"text","text":"Random image from Lorem Picsum, taken at 800x600"}]'
    }
  },
  {
    type: 'text',
    text: 'Hi Jo! Here are the two random images I fetched for you:\n' +
      '\n' +
      '1. The first image features a foggy hillside covered in dense pine trees. The fog creates a misty atmosphere, partially obscuring the tops of the trees. The slope is quite steep and there are patches of snow visible among the trees, adding to the serene and ethereal feel of the scene.\n' +
      '\n' +
      '2. The second image is of a forest with tall, slender pine trees. The forest floor is lush with green undergrowth, giving it a vibrant and lively appearance. The trees are evenly spaced and extend straight up, creating a peaceful and tranquil setting.\n' +
      '\n' +
      'I hope you enjoy these nature-filled visuals!'
  },

---

However, I did notice this when using Azure that if your quota is relatively low, vision requests will hang/error.

[danny-avila]

Trying the same OpenAI trick with Bedrock won't work, either:

ValidationException: A conversation must alternate between user and assistant roles. Make sure the conversation alternates between user and assistant roles and try again.

[dvejsada]

Ok, thanks a lot, @danny-avila. So from practical perspective, it is better for MCP tool to return a link to image rather than an actual image as base64?