Credential Provider Chain returns fatal error

[pbdeuchler]

Describe the bug

I'm encountering some bizarre behavior with the credentials provider chain when trying to perform a GetObject with the s3 sdk. The most common error is a 1 second timeout:

S3 fetch error (other meta): Error
source: Some(DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: ProviderError(ProviderError { source: Unexpected(Unexpected { source: hyper_util::client::legacy::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }) }) }), connection: Unknown } }))
S3 fetch error (other meta): Error
source: Some(DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: ProviderError(ProviderError { source: Unexpected(Unexpected { source: hyper_util::client::legacy::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }) }) }), connection: Unknown } }))
S3 fetch error (other meta): Error
source: Some(DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: ProviderError(ProviderError { source: Unexpected(Unexpected { source: hyper_util::client::legacy::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }) }) }), connection: Unknown } }))

My only guess here is IMDS is timing out? However I also see some total credential provider failures:

S3 fetch error (other meta): Error
source: Some(DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } }))

From what I can tell the IMDS server on my EC2 Instance is operating just fine, and most notably any PutObject calls happily succeed.

Putting an access key and secret in environment variables seems to short circuit the credential provider chain and temporarily fix the bug, but of course this is not a long term solution.

Fwiw I don't expect help debugging this specific issue here, I'm more concerned with the fact that this error path is possible to trigger within the SDK.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

1. Client creation should fail if the entire credential providers chain returns an error, I should not have to make a call first

2. Any network (or otherwise fallible) call involved in the credential chain should be more explicit about what it is trying to reach, and exactly what failed when trying to reach it. Generic timeout errors are frustrating, to say the least.

Current Behavior

(see body for logs)

Reproduction Steps

Current impl (this has been dramatically simplified/hacked to the minimum reproducible bit of code):

async fn client(conf: crate::config::Aws) -> Result<Client, S3ClientError> {
    let region_provider = RegionProviderChain::first_try(Region::new(conf.region().to_owned()));
    let _region = region_provider.region().await.unwrap();
    let shared_config = aws_config::defaults(BehaviorVersion::latest())
        .region(region_provider)
        .load()
        .await;
    Ok(Client::new(&shared_config))
}

    let s3_client = client(conf).await.map_err(S3FetchError::ClientError)?;

    // Get the object from S3
    let response = s3_client
        .get_object()
        .bucket(bucket)
        .key(&s3_key)
        .send()
        .await
        .map_err(|e| match e.into_service_error() {
            aws_sdk_s3::operation::get_object::GetObjectError::NoSuchKey(_) => {
                println!("S3 KEY NOT FOUND");
                S3FetchError::KeyNotFound
            }
            other => {
                println!("S3 fetch error (other meta): {}", other.meta());
                println!("source: {:?}", other.source());
                S3FetchError::GetObjectError(other.into())
            }
        })?;

Obviously it's very unlikely the error is within this s3 specific code, but deep within aws-config and how it interacts with its environment.

Possible Solution

No response

Additional Information/Context

Imho it should not be possible for a credential provider chain to entirely fail on an ec2 instance. At bare minimum any sort of failure to reach the IMDS service or obtain an ec2 instance profile should be explicit and fatal since it's the last step in the chain.

Please, please, please document your credential provider API better in this sdk. I am happy to dive into the innards and mess with the provider chain, or implement my own satisfying the traits, but right now I'm spending my time trying to reverse engineer three different levels of abstraction from the github repo instead of actually fixing the bug in my code.

Last, and I know i'm on my soapbox here, but it's incredibly frustrating to see a deprecated error enum show up in production code. I'm not sure how that happened, but if I can return the error type without invoking a deprecated API then by definition it is not deprecated.

Version

↪  cargo tree | grep aws-
│   ├── aws-config v1.8.0
│   │   ├── aws-credential-types v1.2.3
│   │   │   ├── aws-smithy-async v1.2.5
│   │   │   ├── aws-smithy-runtime-api v1.8.1
│   │   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   │   ├── aws-smithy-types v1.3.2
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-runtime v1.5.8
│   │   │   ├── aws-credential-types v1.2.3 (*)
│   │   │   ├── aws-sigv4 v1.3.3
│   │   │   │   ├── aws-credential-types v1.2.3 (*)
│   │   │   │   ├── aws-smithy-eventstream v0.60.9
│   │   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   │   ├── aws-smithy-http v0.62.1
│   │   │   │   │   ├── aws-smithy-eventstream v0.60.9 (*)
│   │   │   │   │   ├── aws-smithy-runtime-api v1.8.1 (*)
│   │   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.8.1 (*)
│   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.9 (*)
│   │   │   ├── aws-smithy-http v0.62.1 (*)
│   │   │   ├── aws-smithy-runtime v1.8.3
│   │   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   │   ├── aws-smithy-http v0.62.1 (*)
│   │   │   │   ├── aws-smithy-http-client v1.0.5
│   │   │   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   │   │   ├── aws-smithy-runtime-api v1.8.1 (*)
│   │   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   │   │   │   │   ├── aws-lc-rs v1.13.1
│   │   │   │   │   │   │   │   ├── aws-lc-sys v0.29.0
│   │   │   │   │   │   │   │   ├── aws-lc-rs v1.13.1 (*)
│   │   │   │   ├── aws-smithy-observability v0.1.3
│   │   │   │   │   └── aws-smithy-runtime-api v1.8.1 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.8.1 (*)
│   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-smithy-runtime-api v1.8.1 (*)
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-types v1.3.7
│   │   │   │   ├── aws-credential-types v1.2.3 (*)
│   │   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.8.1 (*)
│   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-sdk-sso v1.73.0
│   │   │   ├── aws-credential-types v1.2.3 (*)
│   │   │   ├── aws-runtime v1.5.8 (*)
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-http v0.62.1 (*)
│   │   │   ├── aws-smithy-json v0.61.4
│   │   │   │   └── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-smithy-runtime v1.8.3 (*)
│   │   │   ├── aws-smithy-runtime-api v1.8.1 (*)
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-types v1.3.7 (*)
│   │   ├── aws-sdk-ssooidc v1.74.0
│   │   │   ├── aws-credential-types v1.2.3 (*)
│   │   │   ├── aws-runtime v1.5.8 (*)
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-http v0.62.1 (*)
│   │   │   ├── aws-smithy-json v0.61.4 (*)
│   │   │   ├── aws-smithy-runtime v1.8.3 (*)
│   │   │   ├── aws-smithy-runtime-api v1.8.1 (*)
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-types v1.3.7 (*)
│   │   ├── aws-sdk-sts v1.75.0
│   │   │   ├── aws-credential-types v1.2.3 (*)
│   │   │   ├── aws-runtime v1.5.8 (*)
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-http v0.62.1 (*)
│   │   │   ├── aws-smithy-json v0.61.4 (*)
│   │   │   ├── aws-smithy-query v0.60.7
│   │   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-smithy-runtime v1.8.3 (*)
│   │   │   ├── aws-smithy-runtime-api v1.8.1 (*)
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   │   ├── aws-smithy-xml v0.60.10
│   │   │   ├── aws-types v1.3.7 (*)
│   │   ├── aws-smithy-async v1.2.5 (*)
│   │   ├── aws-smithy-http v0.62.1 (*)
│   │   ├── aws-smithy-json v0.61.4 (*)
│   │   ├── aws-smithy-runtime v1.8.3 (*)
│   │   ├── aws-smithy-runtime-api v1.8.1 (*)
│   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-types v1.3.7 (*)
│   ├── aws-sdk-s3 v1.93.0
│   │   ├── aws-credential-types v1.2.3 (*)
│   │   ├── aws-runtime v1.5.8 (*)
│   │   ├── aws-sigv4 v1.3.3 (*)
│   │   ├── aws-smithy-async v1.2.5 (*)
│   │   ├── aws-smithy-checksums v0.63.3
│   │   │   ├── aws-smithy-http v0.62.1 (*)
│   │   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-smithy-eventstream v0.60.9 (*)
│   │   ├── aws-smithy-http v0.62.1 (*)
│   │   ├── aws-smithy-json v0.61.4 (*)
│   │   ├── aws-smithy-runtime v1.8.3 (*)
│   │   ├── aws-smithy-runtime-api v1.8.1 (*)
│   │   ├── aws-smithy-types v1.3.2 (*)
│   │   ├── aws-smithy-xml v0.60.10 (*)
│   │   ├── aws-types v1.3.7 (*)

Environment details (OS name and version, etc.)

Ubuntu 24.04-arm64

Logs

No response

[ysaito1001]

It seems like you're expecting the IMDS credentials provider within the default credentials chain to return credentials under normal conditions, but you're noticing a potential regression.

This issue may originate from the recent release, especially when you're using aws-config version 1.8.0.

1. Client creation should fail if the entire credential providers chain returns an error, I should not have to make a call first

Credentials are fetched when an operation is called on the client. The client creation only configures the "recipe" for how credentials are retrieved.

2. Any network (or otherwise fallible) call involved in the credential chain should be more explicit about what it is trying to reach, and exactly what failed when trying to reach it. Generic timeout errors are frustrating, to say the least.

We usually propagate errors when encountered (ex. from IMDS), but I noticed the unhandled arm and am wondering if the context might have been lost there.

If you turn on tracing_subscriber at the TRACE level, what errors/anomalies do you see in the trace log that you did not observe when you were using an older aws-config?

[ysaito1001]

it's incredibly frustrating to see a deprecated error enum show up in production code

Could you help point us to an example?