BUG: Bedrock ConverseAPI with Anthropic Claude-3.5-Sonnet

[MiNeves00]

System Info

Using bedrock model anthropic.claude-3-5-sonnet-20241022-v2:0 breaks when doing Sync Stream calls, works fine for: Async Non-Stream, Async Stream, Sync Non-Stream

Error

Message:

  File "/Users/miguelneves/Documents/LLM_Studio/LLMstudio/examples/core.py", line 87, in run_provider
    response_sync_stream = llm.chat(**chat_request)
                           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/miguelneves/Documents/LLM_Studio/LLMstudio/libs/core/llmstudio_core/providers/provider.py", line 304, in chat
    raise ProviderError(str(e))
llmstudio_core.exceptions.ProviderError: An error occurred (ThrottlingException) when calling the ConverseStream operation (reached max retries: 4): Too many requests, please wait before trying again.

Who can help?

No response

Related Components

• API
• SDK
• UI

Reproduction

in the examples/core.py, add the following line at the end and run it with python examples/core.py:

multiple_provider_runs(provider="bedrock", model="anthropic.claude-3-5-sonnet-20241022-v2:0", num_runs=1, api_key=None, region=os.environ["BEDROCK_REGION"], secret_key=os.environ["BEDROCK_SECRET_KEY"], access_key=os.environ["BEDROCK_ACCESS_KEY"])

Expected behavior

Running the examples/core.py for this model should not give any errors.