Bug Report

Description

The AmazonBedrockModel class fails with AccessDeniedException when used in DeepEval's async evaluation context, despite having valid AWS credentials and successful direct Bedrock API access.

Environment

• DeepEval Version: 3.3.5 and 3.3.6 (tested both)
• Python Version: 3.11.11
• AWS Region: eu-west-1
• Model: anthropic.claude-3-haiku-20240307-v1:0

Steps to Reproduce

1. Configure AWS credentials in .env file with valid AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
2. Set up DeepEval evaluation with AmazonBedrockModel:

   model_instance = AmazonBedrockModel(
       model_id="anthropic.claude-3-haiku-20240307-v1:0",
       aws_access_key_id=aws_access_key,
       aws_secret_access_key=aws_secret_key,
       region_name="eu-west-1"
   )

3. Run evaluation using evaluate(test_cases=test_cases, metrics=metrics)

Expected Behavior

Evaluation should proceed successfully using the Anthropic Claude model via AWS Bedrock.

Actual Behavior

botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the Converse operation: You don't have access to the model with the specified model ID.

Additional Context

• ✅ Direct boto3 access works: Same credentials work perfectly with direct boto3.client('bedrock-runtime') calls
• ✅ Model access confirmed: The model is accessible and working via direct API calls
• ✅ Credentials are loaded: Environment variables are properly set and accessible
• ❌ Only fails in DeepEval: The error only occurs when using AmazonBedrockModel within DeepEval's async evaluation context
• ✅ OpenAI models work: Switching to gpt-4o-mini works perfectly with the same evaluation setup

Workaround

Currently using OpenAI models as a workaround, but would prefer to use AWS Bedrock models.

Possible Root Cause

The issue appears to be related to how AWS credentials are handled in the async evaluation context, possibly due to environment variable inheritance or session management in concurrent/async operations.