Stuck on Using Bedrock Invoke API to generate response

[justmekaren]

Package

langchain-aws

Checked other resources

• I added a descriptive title to this issue
• I searched the LangChain documentation with the integrated search
• I used the GitHub search to find a similar issue and didn't find it
• I am sure this is a bug and not a question or request for help

Example Code

from langchain_aws import ChatBedrock

# Local Imports
from libs.utility.clients import _get_client
from libs.utility.logging import logging
import os

default_max_tokens = "Value"
llm_model = "ModelId"


bedrock_client = _get_client("bedrock-runtime", region="us-east-1")

def llm_instance(
    modelId=llm_model, streaming=True, temperature=0.3, max_tokens=default_max_tokens
):
    """Returns an instance of the LLM to use
    :param modelId: The ID of the LLM
    :param streaming: Boolean for instance of the LLM to enable streaming or no
    :param temperature: Creativity level for the LLM
    """
    logging.info(f"Max_tokens: {max_tokens}")
    
    return ChatBedrock(
        model_id=modelId,
        client=bedrock_client,
        streaming=streaming,
        model_kwargs=dict(temperature=temperature, max_tokens=int(max_tokens)),
    )


llm_tool = llm_instance(streaming=False, temperature=0.0)

logging.info("Instance created successfully.")

Error Message and Stack Trace (if applicable)

stuck on "Using Bedrock Invoke API to generate response" on llm invoke

libs/aws/langchain_aws/llms/bedrock.py

Description

I am getting inconsistencies when invoking the model. Cloudwatch logs shows this message "Using Bedrock Invoke API to generate response" but it never returned the LLM response back.

boto3 config max retry 3, read timeout = 120s, connect timeout = 10

I've checked the CW metrics and the model invocation actually did return a response. This issue happened 5/10 attempts.

[ps06756]

For reference,
I have created a wrapper on top of ChatBedrockEnhanced , and have never encountered this issue

Reference

class ChatBedrockEnhanced(ChatBedrock):
    @retry(
        retry=retry_if_exception_type(ClientError),
        wait=wait_exponential(
            multiplier=1,  # bse delay multiplier
            min=4,  # min wait time in seconds
            max=10,  # max wait time in seconds
        ),
        stop=stop_after_attempt(6),
        before_sleep=lambda retry_state: logger.info(
            f"Throttled by AWS. Attempt {retry_state.attempt_number}. Waiting..."
        ),
    )
    def invoke(self, *args, **kwargs):
        try:
            response = super().invoke(*args, **kwargs)
            return response
        except ClientError as e:
            logger.error(f"messages_sent_to_bedrock {str(args)}")
            if e.response["Error"]["Code"] == "ThrottlingException":
                logger.error("AWS Bedrock throttling detected")
                raise
            raise

You can try to use this and see what is happening.
Maybe you are getting throttled by the API ?

Let me know how it goes

[justmekaren]

For reference, I have created a wrapper on top of ChatBedrockEnhanced , and have never encountered this issue

Reference

class ChatBedrockEnhanced(ChatBedrock):
    @retry(
        retry=retry_if_exception_type(ClientError),
        wait=wait_exponential(
            multiplier=1,  # bse delay multiplier
            min=4,  # min wait time in seconds
            max=10,  # max wait time in seconds
        ),
        stop=stop_after_attempt(6),
        before_sleep=lambda retry_state: logger.info(
            f"Throttled by AWS. Attempt {retry_state.attempt_number}. Waiting..."
        ),
    )
    def invoke(self, *args, **kwargs):
        try:
            response = super().invoke(*args, **kwargs)
            return response
        except ClientError as e:
            logger.error(f"messages_sent_to_bedrock {str(args)}")
            if e.response["Error"]["Code"] == "ThrottlingException":
                logger.error("AWS Bedrock throttling detected")
                raise
            raise

You can try to use this and see what is happening. Maybe you are getting throttled by the API ?

Let me know how it goes

I get the same response. I have MCP server hosted in bedrock agentcore runtime. I never had this issue before when I was hosting MCP server in Lambda. I dont see issue or throttling in bedrock invoke_model, in fact all my API calls are getting responded too. Its only the langchain chatbedrock not picking up the response from the invocation. I honestly could not figure out what is happening during that process as this happens intermitently.