Error when accessing Bedrock model in different area

[dvejsada]

What happened?

We use a Bedrock endpoint with a default region of eu-west-3. Models deployed there work fine. However, recently we have added also us-east-1 deployment to get access to Anthropic Claude 3.5 Sonnet v2 and new Amazon models. These however are not working.

Config:
librechat.yaml

  bedrock:
    titleModel: "anthropic.claude-3-haiku-20240307-v1:0"
    availableRegions:
      - "eu-central-1"
      - "eu-west-3"
      - "us-east-1"

.env:

BEDROCK_AWS_DEFAULT_REGION=eu-west-3
BEDROCK_AWS_ACCESS_KEY_ID=*****************************
BEDROCK_AWS_SECRET_ACCESS_KEY=*****************************
BEDROCK_AWS_MODELS=anthropic.claude-3-haiku-20240307-v1:0,eu.anthropic.claude-3-5-sonnet-20240620-v1:0,amazon.nova-pro-v1:0,us.anthropic.claude-3-5-sonnet-20241022-v2:0,us.anthropic.claude-3-5-haiku-20241022-v1:0

Steps to Reproduce

1. Use Bedrock models in EU.
2. Add Bedrock models in US region.

What browsers are you seeing the problem on?

No response

Relevant log output

Console log:
2024-12-15 21:58:03 error: [api/server/controllers/agents/client.js #sendCompletion] Unhandled error type The provided model identifier is invalid.
2024-12-15 21:58:03 error: [handleAbortError] AI response error; aborting request: The provided model identifier is invalid.

Error log:
{"$fault":"client","$metadata":{"attempts":1,"httpStatusCode":400,"requestId":"68c64bd4-544b-42a6-b770-18497e1475a3","totalRetryDelay":0},"level":"error","message":"[api/server/controllers/agents/client.js #sendCompletion] Unhandled error type The provided model identifier is invalid.","name":"ValidationException","pregelTaskId":"0972a331-b45a-55a5-a464-dad9ee7fb4e9","stack":"ValidationException: The provided model identifier is invalid.\n    at de_ValidationExceptionRes (/app/node_modules/@aws-sdk/client-bedrock-runtime/dist-cjs/index.js:1196:21)\n    at de_CommandError (/app/node_modules/@aws-sdk/client-bedrock-runtime/dist-cjs/index.js:1029:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /app/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20\n    at async /app/node_modules/@smithy/core/dist-cjs/index.js:165:18\n    at async /app/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38\n    at async /app/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22\n    at async ChatBedrockConverse._streamResponseChunks (/app/node_modules/@langchain/aws/dist/chat_models.cjs:695:26)\n    at async ChatBedrockConverse._generateUncached (/app/node_modules/@langchain/core/dist/language_models/chat_models.cjs:176:34)\n    at async ChatBedrockConverse.invoke (/app/node_modules/@langchain/core/dist/language_models/chat_models.cjs:60:24)"}
{"$fault":"client","$metadata":{"attempts":1,"httpStatusCode":400,"requestId":"68c64bd4-544b-42a6-b770-18497e1475a3","totalRetryDelay":0},"level":"error","message":"[handleAbortError] AI response error; aborting request: The provided model identifier is invalid.","name":"ValidationException","pregelTaskId":"0972a331-b45a-55a5-a464-dad9ee7fb4e9","stack":"ValidationException: The provided model identifier is invalid.\n    at de_ValidationExceptionRes (/app/node_modules/@aws-sdk/client-bedrock-runtime/dist-cjs/index.js:1196:21)\n    at de_CommandError (/app/node_modules/@aws-sdk/client-bedrock-runtime/dist-cjs/index.js:1029:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /app/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20\n    at async /app/node_modules/@smithy/core/dist-cjs/index.js:165:18\n    at async /app/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38\n    at async /app/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22\n    at async ChatBedrockConverse._streamResponseChunks (/app/node_modules/@langchain/aws/dist/chat_models.cjs:695:26)\n    at async ChatBedrockConverse._generateUncached (/app/node_modules/@langchain/core/dist/language_models/chat_models.cjs:176:34)\n    at async ChatBedrockConverse.invoke (/app/node_modules/@langchain/core/dist/language_models/chat_models.cjs:60:24)"}

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

Both model identifiers work for me (us-west-2):
BEDROCK_AWS_MODELS=us.anthropic.claude-3-5-sonnet-20241022-v2:0,anthropic.claude-3-5-sonnet-20241022-v2:0

However, you may not have the right region selected, as is necessary when using multiple regions.

Otherwise this will be used: BEDROCK_AWS_DEFAULT_REGION=eu-west-3

Non-default region has to be selected from Parameters:

You can simplify this with presets to select the non-default configuration.

[dvejsada]

Both model identifiers work for me (us-west-2): BEDROCK_AWS_MODELS=us.anthropic.claude-3-5-sonnet-20241022-v2:0,anthropic.claude-3-5-sonnet-20241022-v2:0

However, you may not have the right region selected, as is necessary when using multiple regions.

Otherwise this will be used: BEDROCK_AWS_DEFAULT_REGION=eu-west-3

Non-default region has to be selected from Parameters:

You can simplify this with presets to select the non-default configuration.

Thanks! The issue was that you cannot use the cross region interference name of the model, as it disables the selection of the region in model preferences (at least with agent endpoint). This is fine if the cross interference model is in the same region (i.e. EU cross region interference with EU default region), but it will fail in case of US cross region interference with EU default region. Maybe consider to keep the region selection always active in case of all types of AWS models, including cross region interference models.

[dvejsada]

@danny-avila I got it working for Anthropic models as per above, but not for Amazon Nova models as their require use of the cross region interference id, in case of which the region selection is disabled in LibreChat and the model fails (most likely due to fact it will use default EU region).

Based on the above, please consider:

• keep the region selection active also for cross region interference models or
• determine the region automatically not based on default AWS region, but based on model prefix (ie. eu-/us-)

[dvejsada]

@danny-avila I got it working for Anthropic models as per above, but not for Amazon Nova models as their require use of the cross region interference id, in case of which the region selection is disabled in LibreChat and the model fails (most likely due to fact it will use default EU region).

Based on the above, please consider:

• keep the region selection active also for cross region interference models or
• determine the region automatically not based on default AWS region, but based on model prefix (ie. eu-/us-)

Spoke to soon, it is not working again. I cannot switch the region in model preferences in Agents (it is there, but it does not respond to any selection).