"anthropic.claude-3-5-sonnet-20241022-v2:0 with on-demand throughput isn’t supported."

[ldavis9000aws]

What happened?

1. Ran git pull on the repo
2. Ran "docker-compose pull"
3. Restarted Librechat, navigated to the "AWS Bedrock" endpoint, then choose anthropic.claude-3-5-sonnet-20241022-v2:0.
4. When I start a chat, I receive this error:

LibreChat | 2024-10-28 23:01:29 error: [api/server/controllers/agents/client.js #sendCompletion] Unhandled error type Invocation of model ID anthropic.claude-3-5-sonnet-20241022-v2:0 with on-demand throughput isn’t supported. Retry your request with the ID or ARN of an inference profile that contains this model.
LibreChat | 2024-10-28 23:01:29 error: [handleAbortError] AI response error; aborting request: Invocation of model ID anthropic.claude-3-5-sonnet-20241022-v2:0 with on-demand throughput isn’t supported. Retry your request with the ID or ARN of an inference profile that contains this model.

If I switch to the "anthropic.claude-3-5-sonnet-20240620-v1:0" model, it works fine.

Steps to Reproduce

1. Ran a git pull
2. Ran "docker-compose pull"
3. Restarted Librechat, navigated to the "AWS Bedrock" endpoint, then choose anthropic.claude-3-5-sonnet-20241022-v2:0.
4. Start a chat with the new anthropic.claude-3-5-sonnet-20241022-v2:0 model

What browsers are you seeing the problem on?

No response

Relevant log output

No response

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

Moved to troubleshooting since this depends on your configured AWS account/region.

[ldavis9000aws]

It looks like the new Claude 3.5 v2 Sonnet Model in Amazon Bedrock is only available in us-west-2 at the moment. It also required me to duplicate access requests to all of the other models I use via Librechat in us-east-1 for us-west-2 (a region I never use) just to get access to the new model. For now, I guess specifying us-west-2 as the region in the .env is the workaround until it's available in us-east-1. However, if the user were able to specify their AWS creds and region in the endpoint via the LibreChat UI (#4572), we wouldn't have to worry about updating the default region defined in the .env file.

Thanks for pointing me towards the model access issue that was causing the error.

[stephaneminisini]

To be able to use the latest Sonnet 3.5 from the us-east-1 region, the model inference profile id is 'us.anthropic.claude-3-5-sonnet-20241022-v2:0', it will route the inference to the region where the model is hosted.

[arvi18]

Thanks a lot man!

[arvi18]

I was trying to find the modelId for Haiku 3.5 as well
Its in this table below -

Bedrock Dashboard (left sidebar) > Inference > Cross-region inference

[emorgan-korrobio]

when I try this (us.anthropic) I get that I don't have access to the model (I am in US-east-1) and have access to the previous versions of the model as well, is this just something that the admin for our AWS systems has to approve?

[lakshya-kh]

Thank you good Sir.

[MatheusRDG]

thanks!

[dubwub]

Really random, but had to spend a lot of time tracking this down and wanted to put it somewhere (as of 2025/1/9):

• The fix for using us.* allows for us-east-1
• The model seems to be different between the one on us-east-1 and us-west-2 -- we have a usecase involving calling Bedrock with a tool to call an embedded model, which was intermittently failing on us-east-1 but works fine on us-west-2 (which allows the modelID to be * instead of us.*.)
• If anyone else is seeing performance differences, allowing access on us-west-2 and switching modelID to 'anthropic.claude-3-5-sonnet-20241022-v2:0' may be helpful

[lakshya-kh]

Can you please explain performance differences ? Do you mean latency or Quality of output ?

[fdx-apdewis]

I've found that you must specify an inference profile instead of the model ID if using on demand. For APAC apac.anthropic.claude-3-5-sonnet-20240620-v1:0 seems to work in place of the model ID for me. I was encountering the above error prior to this change.

[fdx-apdewis]

See: Cross region inference in the bedrock console.