S3AsyncClient with localstack in test fails to perform S3 operations, uses wrong endpoint

[ofir-popowski]

Type: Bug

Component:
Localstack testcontainer with S3AsyncClient

Describe the bug
Using an @Autowired S3AsyncClient in test environment fails to use S3, possibly hitting wrong endpoint

Sample
To reproduce, I setup an "empty" project, having just awspring's relevant dependencies for S3, and testcontainers. Injecting both S3Client and S3AsyncClient for comparison
(properties file is effectively empty)

dependencies:

implementation(platform("io.awspring.cloud:spring-cloud-aws-dependencies:3.3.0"))
implementation("io.awspring.cloud:spring-cloud-aws-starter")
implementation("io.awspring.cloud:spring-cloud-aws-starter-s3")
implementation("software.amazon.awssdk:aws-crt-client")

setup:

@Testcontainers
@SpringBootTest
class ExAsyncApplicationTests(
    @Autowired private val s3Client: S3Client,
    @Autowired private val s3AsyncClient: S3AsyncClient,
) {

    @Test
    fun s3ClientCreateBucket() {
        val createBucketRequest = CreateBucketRequest.builder().bucket("example-s3-client").build()
        assertDoesNotThrow { s3Client.createBucket(createBucketRequest) }
    }

    @Test
    fun s3AsyncClientCreateBucket() {
        val createBucketRequest = CreateBucketRequest.builder().bucket("example-s3-async").build()
        s3AsyncClient.createBucket(createBucketRequest).join()
    }

    companion object {
        @Container
        @ServiceConnection
        @JvmStatic
        private val localstack = LocalStackContainer(DockerImageName.parse("localstack/localstack"))
    }
}

The S3Client can successfully create a bucket in the localstack container, while the S3AsyncClient has the following exception thrown:

software.amazon.awssdk.services.s3.model.S3Exception: The AWS Access Key Id you provided does not exist in our records.
(Service: S3, Status Code: 403, Request ID: 21K573APH2RKPDJJ,
Extended Request ID: Mnwm35es22gJyJ9sNhG4YTgBiQW13A/uucx5c8sINSeBvle3kvUzjIjL5AzQazDYyjAj4PggwUc=)

I put a breakpoint at io.awspring.cloud.autoconfigure.s3.S3CrtAsyncClientAutoConfiguration#85 to take a look at the credentials provider, and I think it's identical to what I see at io.awspring.cloud.autoconfigure.s3.S3AutoConfiguration#113

Then I added

logging.level.software.amazon=trace
logging.level.io.awspring=trace

It looks like the endpoint is not setup correctly, here's the relevant part from running the s3AsyncClientCreateBucket test:

2025-01-26T22:25:32.404+02:00 TRACE 34440 --- [ex-async] [    Test worker] s.a.a.c.i.ExecutionInterceptorChain      : Old: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=s3.amazonaws.com, encodedPath=, headers=[], queryParameters=[])
New: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=example-s3-async.s3.amazonaws.com, encodedPath=, headers=[], queryParameters=[])

compared to the s3ClientCreateBucket:

2025-01-26T22:26:47.326+02:00 TRACE 35632 --- [ex-async] [    Test worker] s.a.a.c.i.ExecutionInterceptorChain      : Old: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=http, host=127.0.0.1, port=63499, encodedPath=, headers=[], queryParameters=[])
New: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=http, host=127.0.0.1, port=63499, encodedPath=/example-s3-client, headers=[], queryParameters=[])

[maciejwalkowiak]

Thanks for reporting @ofir-popowski. Would you like to submit a fix?

[ofir-popowski]

Yeah sure could give it a try :)

[ofir-popowski]

@maciejwalkowiak opened draft PR #1330