AWS Bedrock returns XML error (Unexpected token <) with Claude 3 Sonnet on v0.8.2

[dedsxc]

What happened?

Description

I am encountering a persistent error when using AWS Bedrock with Claude 3 models (Sonnet/Haiku). The application returns:

Something went wrong. Here's the specific error message we encountered: An error occurred while processing the request: Unexpected token '<', "<?xml vers"... is not valid JSON Deserialization error: to see the raw response, inspect the hidden field {error}.$response on this object.

librechat log

error: [api/server/controllers/agents/client.js #sendCompletion] Operation aborted Unexpected token '<', "<?xml vers"
Deserialization error: to see the raw response, inspect the hidden field {error}.$response on this object.

librechat.yaml

endpoints:
  bedrock:
    enabled: true

Expected behavior
The request should be successfully parsed as JSON. It seems the payload sent by LibreChat v0.8.2 might not be using the required Messages API format for Claude 3, triggering a legacy API error from AWS.

Environment evidence:

• AWS CLI: Success using aws bedrock-runtime invoke-model with the Messages API payload.
• n8n: Success using the same IAM User and Region with Claude 3 Sonnet.

Version Information

v0.8.2@sha256:1e9ffe435b56412f7bf41148673e5594ecb067a3cc3991925d1cc2222db7e2a7

Steps to Reproduce

1. Configure AWS Bedrock in .env using IAM Access/Secret keys.
2. Set BEDROCK_AWS_MODELS=anthropic.claude-3-sonnet-20240229-v1:0,anthropic.claude-3-haiku-20240307-v1:0.
3. Set BEDROCK_AWS_DEFAULT_REGION=us-east-1.
4. Try to send a message using the Claude 3 Sonnet model.

What browsers are you seeing the problem on?

Chrome

Relevant log output

error: [api/server/controllers/agents/client.js #sendCompletion] Operation aborted Unexpected token '<', "<?xml vers"
Deserialization error: to see the raw response, inspect the hidden field {error}.$response on this object.

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[dedsxc]

Ok the issue was an environment variable conflict.

Librechat was prioritizing the global AWS_REGION (used for Minio/S3) over BEDROCK_AWS_DEFAULT_REGION.

[dedsxc]

Need to reopen this issue.

AWS Environment Variables Conflict: Bedrock + S3 MinIO

Description

There's a conflict when using both AWS Bedrock and S3 (MinIO) simultaneously due to AWS SDK credential resolution.

Configuration

• AWS Bedrock: Using BEDROCK_AWS_ACCESS_KEY_ID + BEDROCK_AWS_SECRET_ACCESS_KEY
• S3 MinIO: Using AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY

Issue

When both sets of credentials are configured, the application encounters authentication errors. The AWS SDK appears to prioritize the default AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY variables for all AWS services, including Bedrock, instead of using the Bedrock-specific credentials.

Expected Behavior

The application should:

• Use BEDROCK_AWS_* credentials specifically for Bedrock API calls
• Use AWS_* credentials specifically for S3 operations

Environment

• Using S3-compatible storage (MinIO) with custom endpoint
• Using AWS Bedrock for AI model inference
• Both services require different credentials

[danny-avila]

Thanks for reporting back. This is working as designed, LibreChat's Bedrock client exclusively uses the BEDROCK_AWS_* prefixed environment variables (BEDROCK_AWS_ACCESS_KEY_ID, BEDROCK_AWS_SECRET_ACCESS_KEY, BEDROCK_AWS_DEFAULT_REGION) and passes them directly to the AWS SDK client.

When those BEDROCK_AWS_* credentials are set, they are used and the global AWS_* variables are not read by LibreChat at all.

However, when BEDROCK_AWS_ACCESS_KEY_ID and BEDROCK_AWS_SECRET_ACCESS_KEY are both empty or unset, LibreChat intentionally sets credentials to undefined, which tells the AWS SDK to fall back to its default credential provider chain. That chain checks, in order, AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY env vars, SSO token cache, shared credential files, instance metadata, etc. This is standard AWS SDK behavior, not something LibreChat controls.

The fix for your setup: Make sure you have the BEDROCK_AWS_* prefixed variables set in your .env with your actual Bedrock credentials:

BEDROCK_AWS_DEFAULT_REGION=us-east-1
BEDROCK_AWS_ACCESS_KEY_ID=your_bedrock_access_key
BEDROCK_AWS_SECRET_ACCESS_KEY=your_bedrock_secret_key

This ensures LibreChat passes those credentials directly to the Bedrock client, completely bypassing the default credential chain (and your MinIO AWS_* variables).

Our Bedrock docs cover the credential resolution behavior in detail.

[dedsxc]

Thank you for your quick reply @danny-avila

however, before open the issue, I followed your documentation provided and this is my current setup.

I set both BEDROCK_* var for bedrock and AWS_* var for minio s3 like the following:

env:
    BEDROCK_AWS_DEFAULT_REGION: us-east-1
    BEDROCK_AWS_ACCESS_KEY_ID:
      valueFrom:
        secretKeyRef:
          name: llm-secret
          key: BEDROCK_AWS_ACCESS_KEY_ID
    BEDROCK_AWS_SECRET_ACCESS_KEY:
      valueFrom:
        secretKeyRef:
          name: llm-secret
          key: BEDROCK_AWS_SECRET_ACCESS_KEY

    # S3
    AWS_ENDPOINT_URL: https://minio.local # <---- Or maybe its due to this little one ?
    AWS_DEFAULT_REGION: us-east-1
    AWS_ACCESS_KEY_ID:
      valueFrom:
        secretKeyRef:
          name: minio-distributed
          key: MINIO_ROOT_USER
    AWS_SECRET_ACCESS_KEY:
      valueFrom:
        secretKeyRef:
          name: minio-distributed
          key: MINIO_ROOT_PASSWORD

Then when I try to use librechat this is the log I have:
error: [api/server/controllers/agents/client.js #sendCompletion] Operation aborted Unexpected token '<', "<?xml vers"... is not valid JSON Deserialization error: to see the raw response, inspect the hidden field {error}.$response on this object.

To make it works I need to remove all AWS_* var. So for now I can't use s3 minio as file storage and fallback to local.

Maybe the creds are well separated but what about the AWS_ENDPOINT_URL if it is set to minio endpoint ?

EDIT:
I confirm, when I remove AWS_ENDPOINT_URL its working well, but then I cant use minio as s3 endpoint