Crash in Client Initialization due to possible race condition

[wmorgan6796]

Describe the bug

Issue we're seeing:

 level=INFO time=2025-09-03T19:43:11.748372Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=ClientConfiguration msg="[AWS-DEBUG] ClientConfiguration will use SDK Auto Resolved profile: [default] if not specified by users."
 level=INFO time=2025-09-03T19:43:11.748426Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=FileSystemUtils msg="[AWS-DEBUG] Environment value for variable HOME is /home/nonroot"
 level=INFO time=2025-09-03T19:43:11.748438Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=FileSystemUtils msg="[AWS-DEBUG] Home directory is missing the final / appending one to normalize"
 level=INFO time=2025-09-03T19:43:11.748447Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=FileSystemUtils msg="[AWS-DEBUG] Final Home Directory is /home/nonroot/"
 level=INFO time=2025-09-03T19:43:11.748458Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:97  aws_sdk=Aws::Config::AWSConfigFileProfileConfigLoader msg="Initializing config loader against fileName /home/nonroot/.aws/credentials and using profilePrefix = 0"
 level=INFO time=2025-09-03T19:43:11.748469Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=FileSystemUtils msg="[AWS-DEBUG] Environment value for variable HOME is /home/nonroot"
 level=INFO time=2025-09-03T19:43:11.748479Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=FileSystemUtils msg="[AWS-DEBUG] Home directory is missing the final / appending one to normalize"
 level=INFO time=2025-09-03T19:43:11.748488Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=FileSystemUtils msg="[AWS-DEBUG] Final Home Directory is /home/nonroot/"
 level=INFO time=2025-09-03T19:43:11.748497Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=FileSystemUtils msg="[AWS-DEBUG] Environment value for variable HOME is /home/nonroot"
 level=INFO time=2025-09-03T19:43:11.748506Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=FileSystemUtils msg="[AWS-DEBUG] Home directory is missing the final / appending one to normalize"
 level=INFO time=2025-09-03T19:43:11.748514Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=FileSystemUtils msg="[AWS-DEBUG] Final Home Directory is /home/nonroot/"
 level=INFO time=2025-09-03T19:43:11.748524Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:97  aws_sdk=ProfileConfigFileAWSCredentialsProvider msg="Setting provider to read credentials from /home/nonroot/.aws/credentials for credentials file and /home/nonroot/.aws/config for the config file , for use with profile default"
 level=INFO time=2025-09-03T19:43:11.748535Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:97  aws_sdk=ProcessCredentialsProvider msg="Setting process credentials provider to read config from default"
 level=INFO time=2025-09-03T19:43:11.752858Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=tls-handler msg="[AWS-DEBUG] Set security policy to AWS-CRT-SDK-TLSv1.0-2023 (minimum_tls_version: 128; cipher_pref: 0)\x00"
 level=INFO time=2025-09-03T19:43:11.757905Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=AuthCredentialsProvider msg="[AWS-DEBUG] Successfully loaded all required parameters for sts web identity credentials provider.\x00"
 level=INFO time=2025-09-03T19:43:11.757932Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=channel-bootstrap msg="[AWS-DEBUG] id=0x7f0915b1f8d0: acquiring bootstrap reference\x00"
 level=INFO time=2025-09-03T19:43:11.757945Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:97  aws_sdk=connection-manager msg="id=0x7f0907159800: Successfully created\x00"
 level=INFO time=2025-09-03T19:43:11.758308Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=ClientConfiguration msg="[AWS-DEBUG] Request Compression enabled"
 level=INFO time=2025-09-03T19:43:11.758321Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=ClientConfiguration msg="[AWS-DEBUG] ClientConfiguration will use MinReqCompression: 10240"
 level=INFO time=2025-09-03T19:43:11.758330Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=ClientConfiguration msg="[AWS-DEBUG] ClientConfiguration will use SDK Auto Resolved profile: [default] if not specified by users."
 level=INFO time=2025-09-03T19:43:11.758348Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:97  aws_sdk=SSOBearerTokenProvider msg="Setting sso bearerToken provider to read config from default"
 level=INFO time=2025-09-03T19:43:11.758358Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:97  aws_sdk=SSOCredentialsProvider msg="Setting sso credentials provider to read config from default"
 level=INFO time=2025-09-03T19:43:11.758367Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=ClientConfiguration msg="[AWS-DEBUG] Request Compression enabled"
 level=INFO time=2025-09-03T19:43:11.758378Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=ClientConfiguration msg="[AWS-DEBUG] ClientConfiguration will use MinReqCompression: 10240"
 level=INFO time=2025-09-03T19:43:11.758405Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=ClientConfiguration msg="[AWS-DEBUG] ClientConfiguration will use SDK Auto Resolved profile: [default] if not specified by users."
 level=INFO time=2025-09-03T19:43:11.758429Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=DefaultAWSCredentialsProviderChain msg="[AWS-DEBUG] The environment variable value AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is "
 level=INFO time=2025-09-03T19:43:11.758440Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=DefaultAWSCredentialsProviderChain msg="[AWS-DEBUG] The environment variable value AWS_CONTAINER_CREDENTIALS_FULL_URI is "
 level=INFO time=2025-09-03T19:43:11.758449Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=DefaultAWSCredentialsProviderChain msg="[AWS-DEBUG] The environment variable value AWS_EC2_METADATA_DISABLED is "
 level=INFO time=2025-09-03T19:43:11.758460Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=ClientConfiguration msg="[AWS-DEBUG] Request Compression enabled"
 level=INFO time=2025-09-03T19:43:11.758470Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=ClientConfiguration msg="[AWS-DEBUG] ClientConfiguration will use MinReqCompression: 10240"
 level=INFO time=2025-09-03T19:43:11.758478Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=ClientConfiguration msg="[AWS-DEBUG] ClientConfiguration will use SDK Auto Resolved profile: [default] if not specified by users."
 level=WARNING time=2025-09-03T19:43:11.758497Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:94  aws_sdk=ClientConfiguration msg="User specified profile: [default] is not found, will use the SDK resolved one."
 level=INFO time=2025-09-03T19:43:11.758518Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:97  aws_sdk=EC2MetadataClient msg="Creating AWSHttpResourceClient with max connections 25 and scheme https"
 level=INFO time=2025-09-03T19:43:11.758530Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:97  aws_sdk=CurlHandleContainer msg="Initializing CurlHandleContainer with size 25"
 level=INFO time=2025-09-03T19:43:11.758542Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:97  aws_sdk=InstanceProfileCredentialsProvider msg="Creating Instance with IMDS timeout: 1s, attempts: 1"
 level=INFO time=2025-09-03T19:43:11.758552Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:97  aws_sdk=DefaultAWSCredentialsProviderChain msg="Added EC2 metadata service credentials provider to the provider chain."
 level=INFO time=2025-09-03T19:43:11.758591Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:97  aws_sdk=Aws::Config::AWSConfigFileProfileConfigLoader msg="Unable to open config file /home/nonroot/.aws/credentials for reading."
 level=INFO time=2025-09-03T19:43:11.758602Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:97  aws_sdk=Aws::Config::AWSProfileConfigLoaderBase msg="Failed to reload configuration."
 level=INFO time=2025-09-03T19:43:11.758613Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:97  aws_sdk=ProcessCredentialsProvider msg="Failed to find credential process's profile: default"
 level=INFO time=2025-09-03T19:43:11.758625Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=AuthCredentialsProvider msg="[AWS-DEBUG] id=0x7f0907060ca0: STS_WEB_IDENTITY provider trying to load credentials\x00"
 level=INFO time=2025-09-03T19:43:11.758636Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=connection-manager msg="[AWS-DEBUG] id=0x7f0907159800: Acquire connection\x00"
 level=INFO time=2025-09-03T19:43:11.758649Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=connection-manager msg="[AWS-DEBUG] id=0x7f0907159800: snapshot - state=1, idle_connection_count=0, pending_acquire_count=1, pending_settings_count=0, pending_connect_count=1, vended_connection_count=0, open_connection_count=0, ref_count=1\x00"
 level=INFO time=2025-09-03T19:43:11.758661Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:97  aws_sdk=connection-manager msg="id=0x7f0907159800: Requesting 1 new connections from http\x00"
 level=INFO time=2025-09-03T19:43:11.758673Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=channel-bootstrap msg="[AWS-DEBUG] id=0x7f0915b1f8d0: acquiring bootstrap reference\x00"
 level=INFO time=2025-09-03T19:43:11.758685Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=dns msg="[AWS-DEBUG] id=0x7f09306534c0: Host resolution requested for sts.us-west-1.amazonaws.com\x00"
 level=INFO time=2025-09-03T19:43:11.758696Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=dns msg="[AWS-DEBUG] id=0x7f09306534c0: No cached entries found for sts.us-west-1.amazonaws.com starting new resolver thread.\x00"
 level=INFO time=2025-09-03T19:43:11.758856Z thread_id=0x00007f0900731700 caller=aws_logger.cpp:100  aws_sdk=dns msg="[AWS-DEBUG] static: resolving host sts.us-west-1.amazonaws.com\x00"
 level=INFO time=2025-09-03T19:43:11.766628Z thread_id=0x00007f0900731700 caller=aws_logger.cpp:100  aws_sdk=dns msg="[AWS-DEBUG] static: resolved record: 176.32.114.104\x00"
 level=INFO time=2025-09-03T19:43:11.766666Z thread_id=0x00007f0900731700 caller=aws_logger.cpp:100  aws_sdk=dns msg="[AWS-DEBUG] static, resolving host sts.us-west-1.amazonaws.com successful, returned 1 addresses\x00"
 level=INFO time=2025-09-03T19:43:11.766686Z thread_id=0x00007f0900731700 caller=aws_logger.cpp:100  aws_sdk=dns msg="[AWS-DEBUG] static: new address resolved 176.32.114.104 for host sts.us-west-1.amazonaws.com caching\x00"
 level=INFO time=2025-09-03T19:43:11.766699Z thread_id=0x00007f0900731700 caller=aws_logger.cpp:100  aws_sdk=dns msg="[AWS-DEBUG] static, invoking resolution callback for host sts.us-west-1.amazonaws.com with 1 addresses\x00"
 level=INFO time=2025-09-03T19:43:11.766771Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=socket msg="[AWS-DEBUG] id=0x7f09276a8000 fd=141: initializing with domain 0 and type 0\x00"
 level=INFO time=2025-09-03T19:43:11.766800Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=socket msg="[AWS-DEBUG] id=0x7f09276a8000 fd=141: setting socket options to: keep-alive 0, keep-alive timeout 0, keep-alive interval 0, keep-alive probe count 0.\x00"
 level=INFO time=2025-09-03T19:43:11.766816Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=socket msg="[AWS-DEBUG] id=0x7f09276a8000 fd=141: beginning connect.\x00"
 level=INFO time=2025-09-03T19:43:11.766835Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=socket msg="[AWS-DEBUG] id=0x7f09276a8000 fd=141: connecting to endpoint 176.32.114.104:443.\x00"
 level=INFO time=2025-09-03T19:43:11.768077Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:97  aws_sdk=socket msg="id=0x7f09276a8000 fd=141: connection success\x00"
 level=INFO time=2025-09-03T19:43:11.768096Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=socket msg="[AWS-DEBUG] id=0x7f09276a8000 fd=141: assigning to event loop 0x7f093078b610\x00"
 level=INFO time=2025-09-03T19:43:11.768115Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=channel-bootstrap msg="[AWS-DEBUG] id=0x7f0915b1f8d0: client connection on socket 0x7f09276a8000 completed with error 0 : AWS_ERROR_SUCCESS\x00"
 level=INFO time=2025-09-03T19:43:11.768140Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=channel msg="[AWS-DEBUG] id=0x7f09276be000: Beginning creation and setup of new channel.\x00"
 level=INFO time=2025-09-03T19:43:11.768156Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=channel msg="[AWS-DEBUG] id=0x7f09276be000: setup complete, notifying caller.\x00"
 level=INFO time=2025-09-03T19:43:11.768175Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=channel msg="[AWS-DEBUG] id=0x7f09276be000: no message pool is currently stored in the event-loop local storage, adding 0x7f0927689d00 with max message size 16384, message count 4, with 4 small blocks of 128 bytes.\x00"
 level=INFO time=2025-09-03T19:43:11.768198Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=channel-bootstrap msg="[AWS-DEBUG] id=0x7f0915b1f8d0: channel 0x7f09276be000 setup succeeded: bootstrapping.\x00"
 level=INFO time=2025-09-03T19:43:11.768216Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=socket-handler msg="[AWS-DEBUG] id=0x7f09276dc3c0: Socket handler created with max_read_size of 16384\x00"
 level=INFO time=2025-09-03T19:43:11.770379Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=tls-handler msg="[AWS-DEBUG] id=0x7f09276be300: Remote server name is sts.us-west-1.amazonaws.com\x00"
 level=INFO time=2025-09-03T19:43:11.770403Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=channel-bootstrap msg="[AWS-DEBUG] id=0x7f0915b1f8d0: tls negotiation result 0 on channel 0x7f09276be000\x00"
 level=INFO time=2025-09-03T19:43:11.770418Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:97  aws_sdk=http-connection msg="id=0x7f09276bec00: HTTP/1.1 client connection established.\x00"
 level=INFO time=2025-09-03T19:43:11.770428Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=connection-manager msg="[AWS-DEBUG] id=0x7f0907159800: Received new connection (id=0x7f09276bec00) from http layer\x00"
 level=INFO time=2025-09-03T19:43:11.770439Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=connection-manager msg="[AWS-DEBUG] id=0x7f0907159800: Grabbing pooled connection (0x7f09276bec00)\x00"
 level=INFO time=2025-09-03T19:43:11.770451Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=connection-manager msg="[AWS-DEBUG] id=0x7f0907159800: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=1, ref_count=1\x00"
 level=INFO time=2025-09-03T19:43:11.770463Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=connection-manager msg="[AWS-DEBUG] id=0x7f0907159800: Successfully completed connection acquisition with connection id=0x7f09276bec00\x00"
 level=INFO time=2025-09-03T19:43:11.770562Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=http-stream msg="[AWS-DEBUG] id=0x7f09276bef00: Created client request on connection=0x7f09276bec00: POST / HTTP/1.1\x00"
 level=INFO time=2025-09-03T19:43:11.786429Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=AuthCredentialsProvider msg="[AWS-DEBUG] (id=0x7f0907060ca0) STS_WEB_IDENTITY credentials provider query received http status code 200\x00"
 level=INFO time=2025-09-03T19:43:11.786455Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=http-stream msg="[AWS-DEBUG] id=0x7f09276bef00: Client request complete, response status: 200 (OK).\x00"
 level=INFO time=2025-09-03T19:43:11.786468Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=connection-manager msg="[AWS-DEBUG] id=0x7f0907159800: User releasing connection (id=0x7f09276bec00)\x00"
 level=INFO time=2025-09-03T19:43:11.786480Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=connection-manager msg="[AWS-DEBUG] id=0x7f0907159800: snapshot - state=1, idle_connection_count=1, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=1, ref_count=1\x00"
 level=INFO time=2025-09-03T19:43:11.786510Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:97  aws_sdk=AuthCredentialsProvider msg="(id=0x7f0907060ca0) STS_WEB_IDENTITY credentials provider successfully queried credentials\x00"
 level=INFO time=2025-09-03T19:43:11.786524Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:100  aws_sdk=connection-manager msg="[AWS-DEBUG] id=0x7f0907159800: User releasing connection (id=0x7f09276bec00)\x00"
 level=FATAL time=2025-09-03T19:43:11.786536Z thread_id=0x00007f09245f8700 caller=aws_logger.cpp:91  aws_sdk=connection-manager msg="id=0x7f0907159800: Connection released when vended connection count is zero\x00"
 level=INFO time=2025-09-03T19:43:11.786562Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=AuthCredentialsProvider msg="[AWS-DEBUG] id=0x7f0907060ca0: STS_WEB_IDENTITY provider trying to load credentials\x00"
 level=INFO time=2025-09-03T19:43:11.786608Z thread_id=0x00007f0930abd800 caller=aws_logger.cpp:100  aws_sdk=connection-manager msg="[AWS-DEBUG] id=0x7f0907159800: Acquire connection\x00"
 level=ERROR time=2025-09-03T19:43:11.786620Z thread_id=0x00007f09245f8700 caller=logging.cpp:35  msg="*** Aborted at 1756928591 (unix time) try \"date -d @1756928591\" if you are using GNU date ***\n"
 level=ERROR time=2025-09-03T19:43:11.786990Z thread_id=0x00007f09245f8700 caller=logging.cpp:35  msg="PC: @     0x7f0930ae4602 abort\n"
 level=ERROR time=2025-09-03T19:43:11.787002Z thread_id=0x00007f09245f8700 caller=logging.cpp:35  msg="*** SIGSEGV (@0x0) received by PID 1 (TID 0x7f09245f8700) from PID 0; stack trace: ***\n"

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

AWS SDK works correctly.

Current Behavior

It crashes.

Reproduction Steps

Latest released AWS SDK seems to have the problem

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

1.11.640

Compiler and Version used

GCC 15.2

Operating System and version

Ubuntu 20.04

[sbiscigl]

Reproduction Steps
Latest released AWS SDK seems to have the problem

can you provide any reproduction steps to this? we run nightly tests that include a integration tests for this that we haven't encountered this error with.

additionally with these logs where the segfault actually happens

level=ERROR time=2025-09-03T19:43:11.786620Z thread_id=0x00007f09245f8700 caller=logging.cpp:35 msg="*** Aborted at 1756928591 (unix time) try "date -d @1756928591" if you are using GNU date \n"
level=ERROR time=2025-09-03T19:43:11.786990Z thread_id=0x00007f09245f8700 caller=logging.cpp:35 msg="PC: @ 0x7f0930ae4602 abort\n"
level=ERROR time=2025-09-03T19:43:11.787002Z thread_id=0x00007f09245f8700 caller=logging.cpp:35 msg=" SIGSEGV (@0x0) received by PID 1 (TID 0x7f09245f8700) from PID 0; stack trace: ***\n"

what is logging.cpp, is that your code?

[sbiscigl]

Tried reproducing this in a standalone environment and i cant. Without a minimal reproducible example theres not a lot we can do as theres too many variables with your environment and usage. I did however create a minimal example of it working as expected, if you can update that to fail id be more than happy to jump on it.

Project Structure:

~/sdk-usage-workspace ❯❯❯ tree -L 1
.
├── CMakeLists.txt
├── Dockerfile
├── main.cpp
└── replicate.sh

Dockerfile

FROM public.ecr.aws/ubuntu/ubuntu:22.04

RUN  apt-get update &&\
     apt install -y build-essential git libcurl4-openssl-dev zlib1g-dev libssl-dev cmake

RUN git clone --depth 1 --recurse-submodules https://github.com/aws/aws-sdk-cpp && \
    cd aws-sdk-cpp && \
    mkdir build && \
    cd build && \
    cmake .. -DBUILD_TESTING=OFF -DAUTORUN_UNIT_TESTS=OFF -DBUILD_ONLY="s3" &&\
    cmake --build . && \
    cmake --install .

# Copy over and build
RUN mkdir sdk-example
COPY CMakeLists.txt /sdk-example/CMakeLists.txt
COPY main.cpp /sdk-example/main.cpp
RUN cd sdk-example &&\
    mkdir build &&\
    cd build &&\
    cmake -DCMAKE_PREFIX_PATH=/sdk-install .. && \
    cmake --build .

CMakeLists.txt

cmake_minimum_required(VERSION 3.13)
project(sdk_usage_workspace)
set(CMAKE_CXX_STANDARD 20)
find_package(AWSSDK REQUIRED COMPONENTS s3)
add_executable(${PROJECT_NAME} main.cpp)
target_link_libraries(${PROJECT_NAME} PRIVATE ${AWSSDK_LINK_LIBRARIES})

main.cpp

#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>

using namespace Aws;
using namespace Aws::S3;

class AwsSdkContext {
 public:
  explicit AwsSdkContext(SDKOptions&& options) : options_(options) { InitAPI(options_); }
  ~AwsSdkContext() { ShutdownAPI(options_); }

 private:
  SDKOptions options_;
};

auto main() -> int {
  SDKOptions options{};
  options.loggingOptions.logLevel = Utils::Logging::LogLevel::Debug;
  AwsSdkContext context{std::move(options)};
  S3Client client{};
  return 0;
}

replicate.sh

set -u

# build image
docker build -t test-image .

# run example
docker run --name test-image test-image /sdk-example/build/sdk_usage_workspace

when we run with ./replicate.sh we see the sample application exit successfully (although it may take a moment)

If you can update this to fail, will be happy to jump on it.

[wmorgan6796]

So I believe I’ve identified the commit which caused this:

c6ed3e3

Unfortunately I’m unable to share a lot of code due to internal policy and NDA. Would filing a support ticket through our TAM be better?

[wmorgan6796]

We’re using GCC 15.2 with the latest curl release on Ubuntu 20.04.

I’m not entirely sure what more I’m allowed to share.

[sbiscigl]

So I believe I’ve identified the commit which caused this:
c6ed3e3

so that commit changes the implementation yes, which does change something, but I really dont know how it impacts you unless i see how you are using the SDK.

Unfortunately I’m unable to share a lot of code due to internal policy and NDA. Would filing a support ticket through our TAM be better?

I would prefer a minimal example, debugging your code directly isn't something that we can support. Even if you reach out through a TAM I will likely still tell you, you need a minimal reproducible example.

We’re using GCC 15.2 with the latest curl release on Ubuntu 20.04.

yep thats reflected in the docker image

additionally i tried the same thing if the client uses a STSAssumeRoleWebIdentityCredentialsProvider and i still see it working as expected, full example:

Project Structure:

~/sdk-usage-workspace ❯❯❯ tree -L 1
.
├── CMakeLists.txt
├── Dockerfile
├── main.cpp
└── replicate.sh

Dockerfile

FROM public.ecr.aws/ubuntu/ubuntu:22.04

RUN  apt-get update &&\
     apt install -y build-essential git libcurl4-openssl-dev zlib1g-dev libssl-dev cmake

RUN git clone --depth 1 --recurse-submodules https://github.com/aws/aws-sdk-cpp && \
    cd aws-sdk-cpp && \
    mkdir build && \
    cd build && \
    cmake .. -DBUILD_TESTING=OFF -DAUTORUN_UNIT_TESTS=OFF -DBUILD_ONLY="sts;iam;cognito-identity;s3" &&\
    cmake --build . && \
    cmake --install .

# Copy over and build
RUN mkdir sdk-example
COPY CMakeLists.txt /sdk-example/CMakeLists.txt
COPY main.cpp /sdk-example/main.cpp
RUN cd sdk-example &&\
    mkdir build &&\
    cd build &&\
    cmake -DCMAKE_PREFIX_PATH=/sdk-install .. && \
    cmake --build .

CMakeLists.txt

cmake_minimum_required(VERSION 3.13)
project(sdk_usage_workspace)
set(CMAKE_CXX_STANDARD 20)
find_package(AWSSDK REQUIRED COMPONENTS sts;iam;cognito-identity;s3)
add_executable(${PROJECT_NAME} main.cpp)
target_link_libraries(${PROJECT_NAME} PRIVATE ${AWSSDK_LINK_LIBRARIES})

main.cpp

#include <aws/cognito-identity/CognitoIdentityClient.h>
#include <aws/cognito-identity/model/CreateIdentityPoolRequest.h>
#include <aws/cognito-identity/model/DeleteIdentityPoolRequest.h>
#include <aws/cognito-identity/model/GetIdRequest.h>
#include <aws/cognito-identity/model/GetOpenIdTokenRequest.h>
#include <aws/cognito-identity/model/SetIdentityPoolRolesRequest.h>
#include <aws/core/Aws.h>
#include <aws/core/auth/STSCredentialsProvider.h>
#include <aws/core/utils/FileSystemUtils.h>
#include <aws/core/platform/Environment.h>
#include <aws/iam/IAMClient.h>
#include <aws/iam/model/CreateRoleRequest.h>
#include <aws/iam/model/DeleteRolePolicyRequest.h>
#include <aws/iam/model/DeleteRoleRequest.h>
#include <aws/iam/model/PutRolePolicyRequest.h>
#include <aws/s3/S3Client.h>
#include <aws/sts/STSClient.h>
#include <aws/sts/model/AssumeRoleRequest.h>

using namespace Aws;
using namespace Aws::Client;
using namespace Aws::Auth;
using namespace Aws::Utils;
using namespace Aws::IAM;
using namespace Aws::IAM::Model;
using namespace Aws::S3;
using namespace Aws::STS;
using namespace Aws::STS::Model;
using namespace Aws::CognitoIdentity;
using namespace Aws::CognitoIdentity::Model;

namespace {
const char* TEST_LOG_NAME = "client-crash";
const char* TRUST_POLICY = R"({"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Federated":"cognito-identity.amazonaws.com"},"Action":"sts:AssumeRoleWithWebIdentity","Condition":{"StringEquals":{"cognito-identity.amazonaws.com:aud":"IDENTITY_POOL_ID_PLACEHOLDER"},"ForAnyValue:StringLike":{"cognito-identity.amazonaws.com:amr":"unauthenticated"}}},{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::TEST_ACCOUNT_ID:root"},"Action":"sts:AssumeRole"}]})";
const char* ALLOW_S3_POLICY = R"({"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:List*"],"Resource":["*"]}]})";
// If it takes longer than 60s for a IAM role to be consistent -- thats a problem
// and we likely want to move on and fail the test
const std::chrono::milliseconds IAM_CONSISTENCY_SLEEP = std::chrono::seconds(1);
const size_t MAX_IAM_CONSISTENCY_RETRIES = 60;
}

class CognitoIdentitySetup {
 public:
  CognitoIdentitySetup(const Aws::String& testIdentifier)
      : m_tokenFile{std::ios_base::out | std::ios_base::trunc},
        m_roleName{"WebIdentityTestRole" + testIdentifier},
        m_policyName{"WebIdentityTestPolicy" + testIdentifier},
        m_poolName{"WebIdentityTestPool" + testIdentifier} {
    const auto identityPoolResponse = m_cognitoIdentityClient.CreateIdentityPool(CreateIdentityPoolRequest{}
      .WithIdentityPoolName(m_poolName)
      .WithAllowUnauthenticatedIdentities(true));
    m_identityPoolId = identityPoolResponse.GetResult().GetIdentityPoolId();
    Aws::String trustPolicyWithID{TRUST_POLICY};
    Aws::Utils::StringUtils::Replace(trustPolicyWithID, "IDENTITY_POOL_ID_PLACEHOLDER", m_identityPoolId.c_str());
    Aws::Utils::StringUtils::Replace(trustPolicyWithID, "TEST_ACCOUNT_ID", Aws::Environment::GetEnv("AWS_ACCOUNT_ID").c_str());

    const auto createRoleResponse = m_iamClient.CreateRole(CreateRoleRequest{}
      .WithRoleName(m_roleName)
      .WithAssumeRolePolicyDocument(trustPolicyWithID));
    m_roleArn = createRoleResponse.GetResult().GetRole().GetArn();

    const auto updateRoleResponse = m_iamClient.PutRolePolicy(PutRolePolicyRequest{}
      .WithRoleName(m_roleName)
      .WithPolicyName(m_policyName)
      .WithPolicyDocument(ALLOW_S3_POLICY));

    // Force read after write consistency
    AssumeRoleOutcome assumeRoleResponse;
    size_t attempts = 0;
    bool shouldSleep = false;
    do {
      if (shouldSleep) {
        std::this_thread::sleep_for(IAM_CONSISTENCY_SLEEP);
      }
      assumeRoleResponse = m_stsClient.AssumeRole(AssumeRoleRequest{}
        .WithRoleArn(m_roleArn)
        .WithRoleSessionName("test"));
      shouldSleep = true;
    } while (!assumeRoleResponse.IsSuccess() && attempts++ < MAX_IAM_CONSISTENCY_RETRIES);

    const auto setIdentityPoolRulesResponse = m_cognitoIdentityClient.SetIdentityPoolRoles(SetIdentityPoolRolesRequest{}
      .WithIdentityPoolId(m_identityPoolId)
      .WithRoles({
        {"unauthenticated", m_roleArn}
      }));

    const auto getPoolIdResponse = m_cognitoIdentityClient.GetId(GetIdRequest{}.WithIdentityPoolId(m_identityPoolId));

    const auto identityId = getPoolIdResponse.GetResult().GetIdentityId();

    const auto getTokenResponse = m_cognitoIdentityClient.GetOpenIdToken(GetOpenIdTokenRequest{}.WithIdentityId(identityId));

    const auto token = getTokenResponse.GetResult().GetToken();
    m_tokenFile << token << std::endl;
    m_tokenFile.close();
  }

  ~CognitoIdentitySetup() {
    const auto deletePoolResponse = m_cognitoIdentityClient.DeleteIdentityPool(DeleteIdentityPoolRequest{}
      .WithIdentityPoolId(m_identityPoolId));

    const auto deleteRolePolicyResponse = m_iamClient.DeleteRolePolicy(DeleteRolePolicyRequest{}
      .WithRoleName(m_roleName)
      .WithPolicyName(m_policyName));

    const auto deleteRoleResponse = m_iamClient.DeleteRole(DeleteRoleRequest{}.WithRoleName(m_roleName));
  }

  Aws::String GetRoleArn() {
    return m_roleArn;
  }

  Aws::String GetTokenFileName() {
    return m_tokenFile.GetFileName();
  }

 private:
  IAMClient m_iamClient;
  STSClient m_stsClient;
  CognitoIdentityClient m_cognitoIdentityClient;
  TempFile m_tokenFile;
  Aws::String m_roleName;
  Aws::String m_policyName;
  Aws::String m_poolName;
  Aws::String m_roleArn;
  Aws::String m_identityPoolId;
};

class AwsSdkContext {
 public:
  explicit AwsSdkContext(SDKOptions&& options) : options_(options) { InitAPI(options_); }
  ~AwsSdkContext() { ShutdownAPI(options_); }

 private:
  SDKOptions options_;
};

auto main() -> int {
  SDKOptions options{};
  options.loggingOptions.logLevel = Utils::Logging::LogLevel::Debug;
  AwsSdkContext context{std::move(options)};
  {
    CognitoIdentitySetup testResourcesRAII{UUID::RandomUUID()};
    ClientConfiguration config;
    config.credentialProviderConfig.region = config.region;
    config.credentialProviderConfig.stsCredentialsProviderConfig.roleArn = testResourcesRAII.GetRoleArn();
    config.credentialProviderConfig.stsCredentialsProviderConfig.tokenFilePath = testResourcesRAII.GetTokenFileName();
    auto credsProvider = Aws::MakeShared<STSAssumeRoleWebIdentityCredentialsProvider>(TEST_LOG_NAME, config.credentialProviderConfig);
    S3Client client{std::move(credsProvider)};
    const auto response = client.ListBuckets();
    if (response.IsSuccess()) {
      std::cout << "Success" << std::endl;
    }
  }
  return 0;
}

replicate.sh

#!/bin/zsh

set -u

# build image
docker build -t test-image .

# TODO: you need to set these variables to drive the test

# run example
docker run \
    -e AWS_ACCOUNT_ID=${AWS_ACCOUNT_ID} \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} \
    --name test-image test-image /sdk-example/build/sdk_usage_workspace

If you can update this example would be more than happy to look at it.

[sbiscigl]

one thing from your logs i do notice is we are hitting this case in the CRT

    /* We're probably hosed in this case, but let's not underflow */
    if (manager->internal_ref[AWS_HCMCT_VENDED_CONNECTION] == 0) {
        AWS_LOGF_FATAL(
            AWS_LS_HTTP_CONNECTION_MANAGER,
            "id=%p: Connection released when vended connection count is zero",
            (void *)manager);
        aws_raise_error(AWS_ERROR_HTTP_CONNECTION_MANAGER_VENDED_CONNECTION_UNDERFLOW);
        goto release;
    }

[wmorgan6796]

We do initialize an S3 client 2x (once it’s configured with minio and once with s3) is it possible there is some global state in the CRT getting corrupted by creating 2 S3 objects?

[wmorgan6796]

I know the CRT has some global state in weird places so it wouldn’t surprise me if that was the case.

[sbiscigl]

let me update the example to run with 2x clients/initialization under tsan in the morning to see if we get warnings

[wmorgan6796]

Yeah the only difference then is we then do a Get on the S3Client that is connected to S3