aws_config loading stuck in a docker container

[MathieuDuponchelle]

Describe the bug

I'm sorry if this is a bit of a vague report, but I have no idea where to start looking.

In the GStreamer wrapper for AWS translate, we load configs with:

        let _enter_guard = RUNTIME.enter();

        let config_loader = match (access_key, secret_access_key) {
            (Some(key), Some(secret_key)) => {
                gst::log!(CAT, imp = self, "Using settings credentials");
                aws_config::defaults(*AWS_BEHAVIOR_VERSION).credentials_provider(
                    aws_sdk_translate::config::Credentials::new(
                        key,
                        secret_key,
                        session_token,
                        None,
                        "translate",
                    ),
                )
            }
            _ => {
                gst::log!(CAT, imp = self, "Attempting to get credentials from env...");
                aws_config::defaults(*AWS_BEHAVIOR_VERSION)
            }
        };

        let config_loader = config_loader.region(
            aws_config::meta::region::RegionProviderChain::default_provider()
                .or_else(DEFAULT_REGION),
        );
                                                                                                                                                                                                                                                     
        let config_loader =                                                                                                                                                                                                                          
            config_loader.stalled_stream_protection(StalledStreamProtectionConfig::disabled());                                                                                                                                                      
                                                                                                                                                                                                                                                     
        let config = futures::executor::block_on(config_loader.load());                                                                                                                                                                              
        gst::log!(CAT, imp = self, "Using region {}", config.region().unwrap());

The block_on call usually returns near instantly, but when the same code is used in a docker container started with:

docker run --network host --privileged --gpus all --security-opt=label=disable -e AWS_ACCESS_KEY_ID="XXX" -e AWS_SECRET_ACCESS_KEY="XXX" -it --rm image_name

block_on starts systematically returning after exactly one second, then after a few successful connections eventually ends up never returning.

An interesting bit in the logs is:

2025-04-11T12:50:31.201349Z DEBUG region_provider_chain{provider=DefaultRegionChain(RegionProviderChain { providers: [EnvironmentVariableRegionProvider { env: Env(Real) }, ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), time_source: SharedTimeSource(SystemTimeSource), http_client: None, sleep_impl: Some(SharedAsyncSleep(TokioSleep)), region: None, use_fips: None, use_dual_stack: None, profile_name_override: None } }, ImdsRegionProvider { client: "IMDS client truncated for readability", env: Env(Real) }] })}:imds_load_region:invoke{rpc.service="imds" rpc.method="get"}:try_op:try_attempt{attempt=1}:invoke{rpc.service="imds" rpc.method="get-token"}:try_op:try_attempt{attempt=1}: hyper_util::client::legacy::connect::http: connecting to 169.254.169.254:80
2025-04-11T12:50:32.202310Z TRACE region_provider_chain{provider=DefaultRegionChain(RegionProviderChain { providers: [EnvironmentVariableRegionProvider { env: Env(Real) }, ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), time_source: SharedTimeSource(SystemTimeSource), http_client: None, sleep_impl: Some(SharedAsyncSleep(TokioSleep)), region: None, use_fips: None, use_dual_stack: None, profile_name_override: None } }, ImdsRegionProvider { client: "IMDS client truncated for readability", env: Env(Real) }] })}:imds_load_region:invoke{rpc.service="imds" rpc.method="get"}:try_op:try_attempt{attempt=1}:invoke{rpc.service="imds" rpc.method="get-token"}:try_op:try_attempt{attempt=1}: hyper_util::client::legacy::pool: checkout dropped for ("http", 169.254.169.254)

This is where exactly one second is spent

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I expect config_loader.load() to not block for exactly one second

Current Behavior

config_loader.load() blocks a few times for exactly one second, then ends up never returning

Reproduction Steps

See description, I don't have a standalone test case

Possible Solution

No response

Additional Information/Context

No response

Version

│   ├── aws-config v1.6.1
│   │   ├── aws-credential-types v1.2.2
│   │   │   ├── aws-smithy-async v1.2.5
│   │   │   ├── aws-smithy-runtime-api v1.7.4
│   │   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   │   ├── aws-smithy-types v1.3.0
│   │   │   ├── aws-smithy-types v1.3.0 (*)
│   │   ├── aws-runtime v1.5.6
│   │   │   ├── aws-credential-types v1.2.2 (*)
│   │   │   ├── aws-sigv4 v1.3.0
│   │   │   │   ├── aws-credential-types v1.2.2 (*)
│   │   │   │   ├── aws-smithy-eventstream v0.60.8
│   │   │   │   │   ├── aws-smithy-types v1.3.0 (*)
│   │   │   │   ├── aws-smithy-http v0.62.0
│   │   │   │   │   ├── aws-smithy-eventstream v0.60.8 (*)
│   │   │   │   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   │   │   │   ├── aws-smithy-types v1.3.0 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   │   │   ├── aws-smithy-types v1.3.0 (*)
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.8 (*)
│   │   │   ├── aws-smithy-http v0.62.0 (*)
│   │   │   ├── aws-smithy-runtime v1.8.1
│   │   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   │   ├── aws-smithy-http v0.62.0 (*)
│   │   │   │   ├── aws-smithy-http-client v1.0.1
│   │   │   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   │   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   │   │   │   ├── aws-smithy-types v1.3.0 (*)
│   │   │   │   │   │   │   ├── aws-lc-rs v1.13.0
│   │   │   │   │   │   │   │   ├── aws-lc-sys v0.28.0
│   │   │   │   │   │   │   │   ├── aws-lc-rs v1.13.0 (*)
│   │   │   │   ├── aws-smithy-observability v0.1.2
│   │   │   │   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   │   │   ├── aws-smithy-types v1.3.0 (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   │   ├── aws-smithy-types v1.3.0 (*)
│   │   │   ├── aws-types v1.3.6
│   │   │   │   ├── aws-credential-types v1.2.2 (*)
│   │   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   │   │   ├── aws-smithy-types v1.3.0 (*)
│   │   ├── aws-sdk-sso v1.64.0
│   │   │   ├── aws-credential-types v1.2.2 (*)
│   │   │   ├── aws-runtime v1.5.6 (*)
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-http v0.62.0 (*)
│   │   │   ├── aws-smithy-json v0.61.3
│   │   │   │   └── aws-smithy-types v1.3.0 (*)
│   │   │   ├── aws-smithy-runtime v1.8.1 (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   │   ├── aws-smithy-types v1.3.0 (*)
│   │   │   ├── aws-types v1.3.6 (*)
│   │   ├── aws-sdk-ssooidc v1.65.0
│   │   │   ├── aws-credential-types v1.2.2 (*)
│   │   │   ├── aws-runtime v1.5.6 (*)
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-http v0.62.0 (*)
│   │   │   ├── aws-smithy-json v0.61.3 (*)
│   │   │   ├── aws-smithy-runtime v1.8.1 (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   │   ├── aws-smithy-types v1.3.0 (*)
│   │   │   ├── aws-types v1.3.6 (*)
│   │   ├── aws-sdk-sts v1.65.0
│   │   │   ├── aws-credential-types v1.2.2 (*)
│   │   │   ├── aws-runtime v1.5.6 (*)
│   │   │   ├── aws-smithy-async v1.2.5 (*)
│   │   │   ├── aws-smithy-http v0.62.0 (*)
│   │   │   ├── aws-smithy-json v0.61.3 (*)
│   │   │   ├── aws-smithy-query v0.60.7
│   │   │   │   ├── aws-smithy-types v1.3.0 (*)
│   │   │   ├── aws-smithy-runtime v1.8.1 (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   │   ├── aws-smithy-types v1.3.0 (*)
│   │   │   ├── aws-smithy-xml v0.60.9
│   │   │   ├── aws-types v1.3.6 (*)
│   │   ├── aws-smithy-async v1.2.5 (*)
│   │   ├── aws-smithy-http v0.62.0 (*)
│   │   ├── aws-smithy-json v0.61.3 (*)
│   │   ├── aws-smithy-runtime v1.8.1 (*)
│   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   ├── aws-smithy-types v1.3.0 (*)
│   │   ├── aws-types v1.3.6 (*)
│   ├── aws-credential-types v1.2.2 (*)
│   ├── aws-sdk-polly v1.68.0
│   │   ├── aws-credential-types v1.2.2 (*)
│   │   ├── aws-runtime v1.5.6 (*)
│   │   ├── aws-sigv4 v1.3.0 (*)
│   │   ├── aws-smithy-async v1.2.5 (*)
│   │   ├── aws-smithy-http v0.62.0 (*)
│   │   ├── aws-smithy-json v0.61.3 (*)
│   │   ├── aws-smithy-runtime v1.8.1 (*)
│   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   ├── aws-smithy-types v1.3.0 (*)
│   │   ├── aws-types v1.3.6 (*)
│   ├── aws-sdk-s3 v1.82.0
│   │   ├── aws-credential-types v1.2.2 (*)
│   │   ├── aws-runtime v1.5.6 (*)
│   │   ├── aws-sigv4 v1.3.0 (*)
│   │   ├── aws-smithy-async v1.2.5 (*)
│   │   ├── aws-smithy-checksums v0.63.1
│   │   │   ├── aws-smithy-http v0.62.0 (*)
│   │   │   ├── aws-smithy-types v1.3.0 (*)
│   │   ├── aws-smithy-eventstream v0.60.8 (*)
│   │   ├── aws-smithy-http v0.62.0 (*)
│   │   ├── aws-smithy-json v0.61.3 (*)
│   │   ├── aws-smithy-runtime v1.8.1 (*)
│   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   ├── aws-smithy-types v1.3.0 (*)
│   │   ├── aws-smithy-xml v0.60.9 (*)
│   │   ├── aws-types v1.3.6 (*)
│   ├── aws-sdk-transcribestreaming v1.66.0
│   │   ├── aws-credential-types v1.2.2 (*)
│   │   ├── aws-runtime v1.5.6 (*)
│   │   ├── aws-sigv4 v1.3.0 (*)
│   │   ├── aws-smithy-async v1.2.5 (*)
│   │   ├── aws-smithy-eventstream v0.60.8 (*)
│   │   ├── aws-smithy-http v0.62.0 (*)
│   │   ├── aws-smithy-json v0.61.3 (*)
│   │   ├── aws-smithy-runtime v1.8.1 (*)
│   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   ├── aws-smithy-types v1.3.0 (*)
│   │   ├── aws-types v1.3.6 (*)
│   ├── aws-sdk-translate v1.64.0
│   │   ├── aws-credential-types v1.2.2 (*)
│   │   ├── aws-runtime v1.5.6 (*)
│   │   ├── aws-smithy-async v1.2.5 (*)
│   │   ├── aws-smithy-http v0.62.0 (*)
│   │   ├── aws-smithy-json v0.61.3 (*)
│   │   ├── aws-smithy-runtime v1.8.1 (*)
│   │   ├── aws-smithy-runtime-api v1.7.4 (*)
│   │   ├── aws-smithy-types v1.3.0 (*)
│   │   ├── aws-types v1.3.6 (*)
│   ├── aws-types v1.3.6 (*)

Environment details (OS name and version, etc.)

Fedora 40 host / ubuntu 24.04 docker container

Logs

logs.txt

[MathieuDuponchelle]

I identified the difference between my host and the container: I have a ~/.aws/config file on my machine that contains:

[default]
region = us-east-1

With it, config loading is instant, without it stays stuck for one second attempting to connect to a specific address (http://169.254.169.254/). Not sure if this is intended behavior.

[MathieuDuponchelle]

AWS_EC2_METADATA_DISABLED=true addresses this

[MathieuDuponchelle]

Closing as I have a solution

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[landonxjames]

For anyone encountering this in the future. That IP address 169.254.169.254 is for the EC2 IMDS endpoints. The reason it is reaching out there is because it could not find a region specified elsewhere. The logic for region selection is available in our docs. To get around this you can either specify a region in a way that is picked up earlier in the default region provider chain, or explicitly provide your own implementation of ProvideRegion. Depending on your usecase this could be as simple as providing an explicit region from a string.

[MathieuDuponchelle]

@landonxjames thanks for the more detailed explanation, I figured I had at least not done an xkcd 979 here :)

Re specifying a region earlier in the chain, my approach for those elements has been to just provide a default region as a final fallback and leave the default chain otherwise alone. Is there a way to explicitly disable IMDS from the API by the way? The environment variable is fine for my use case at the moment, but I might want to expose a property in the elements for opting-into this behavior.