S3 file upload fails with HTTP 400 (upload to non-amazon S3 storage)

[brake]

Type: Bug

Component: S3

Describe the bug
After upgrading project to version 3.4.0 receive HTTP 400.
TLDR - downgrade to 3.3.0 fixes the issue.

log

io.awspring.cloud.s3.S3Exception: Simple upload failed.
        at io.awspring.cloud.s3.InMemoryBufferingS3OutputStream.putObject(InMemoryBufferingS3OutputStream.java:237)
        at io.awspring.cloud.s3.InMemoryBufferingS3OutputStream.close(InMemoryBufferingS3OutputStream.java:144)
        at java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:190)
        at java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:190)
        at kotlin.io.CloseableKt.closeFinally(Closeable.kt:56)

<skip>

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: (Service: S3, Status Code: 400, Request ID: tx000000000000010eb9d1b-0068b91793-8b25221-default) (SDK Attempt Count: 1)
        at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:113)
        at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:61)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
        at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:11227)
        at io.awspring.cloud.s3.InMemoryBufferingS3OutputStream.putObject(InMemoryBufferingS3OutputStream.java:227)

application.yml:

  cloud.aws:
    credentials:
      access-key: ${S3_ACCESS_KEY}
      secret-key: ${S3_SECRET_KEY}
    s3:
      endpoint: ${S3_ENDPOINT}
      path-style-access-enabled: true
      region: unknown

Only customized configuration is:

    @Bean
    fun s3ClientCustomizer(props: ApplicationProperties): AwsSyncClientCustomizer =
        AwsSyncClientCustomizer { builder ->
            builder.httpClient(
                ApacheHttpClient
                    .builder()
                    .connectionTimeout(props.awsSyncHttpClient.connectTimeout)
                    .socketTimeout(props.awsSyncHttpClient.socketTimeout)
                    .build()
            )

There no other awspring customizations in project, only timeouts for http-client.

build.gradle dependencies

awsVersion here is actually awspring version.

    implementation platform("io.awspring.cloud:spring-cloud-aws-dependencies:$awsVersion")
    implementation "io.awspring.cloud:spring-cloud-aws-starter-s3:$awsVersion"
    implementation "software.amazon.awssdk:apache-client"

Sample
After downgrade to version 3.3.0 everything is working fine, without of some code or configuration file changes.

[maciejwalkowiak]

Do you know exactly what's the issue and are you interested in contributing a fix?

What non-amazon storage is it? Can you provide a sample project that reproduces it?

[brake]

What non-amazon storage is it?

https://docs.ceph.com/en/latest/radosgw/s3/

[ooraini]

Similar issue with OCI, getting a 403 after upgrading