aws sdk 1.12.780 generate 403 error

[kel-web]

Describe the bug

Hi there,

We are using OTEL java agent to instrument our java Application. We have an 403 error with AWS S3 library.
It's the same issue #9452

Full stacktrace :

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: null; S3 Extended Request ID: null; Proxy: proxy-srv.prod.svc.darva.prive), S3 Extended Request ID: null
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1912)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1450)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1419)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1183)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:838)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:805)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:779)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:735)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:717)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:581)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5605)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5552)
	at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:4196)
	at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1329)
	at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1319)
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExistV2(AmazonS3Client.java:1457)

AWS S3 version :

 <dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>aws-java-sdk-s3</artifactId>
      <version>1.12.780</version>
  </dependency>

Tested with Java agent versions 2.10.0 and 2.11.0

The only workaround that works for us is to disable the OTEL AWS-SDK by starting the JVM with the parameters

-Dotel.instrumentation.aws-sdk.enabled=false

Steps to reproduce

Use aws-java-sdk-s3 to put an object in a bucket

Expected behavior

Should process without exceptions

Actual behavior

Causes 403 auth error :

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: null; S3 Extended Request ID: null; Proxy: proxy-srv.prod.svc.darva.prive), S3 Extended Request ID: null
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1912)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1450)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1419)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1183)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:838)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:805)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:779)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:735)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:717)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:581)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5605)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5552)
	at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:4196)
	at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1329)
	at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1319)
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExistV2(AmazonS3Client.java:1457)

Javaagent or library instrumentation version

2.10.0, 2.11.0

Environment

JDK: Eclipse Adoptium OpenJDK 64-Bit Server VM 21.0.5+11-LTS
OS: All plateforms

Additional context

No response

[laurit]

Did this work with a previous version of the aws sdk? Have you tried does it only break with doesBucketExistV2 api or with other apis too?

[kel-web]

Hi @laurit,

Tested with versions :

• 1.12.800
• 1.12.707
• 1.12.406

And the error occurs also when calling the PUT method :

com.amazonaws.retry.ClockSkewAdjuster;;getServerDate;;;Reported server date (from 'Date' header): Wed, 08 Jan 2025 15:31:29 GMT
com.amazonaws.http.AmazonHttpClient$RequestExecutor;;handleErrorResponse;;;Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: null; S3 Extended Request ID: null; Proxy: xxx), S3 Extended Request ID: null
com.amazonaws.retry.ClockSkewAdjuster;;getServerDate;;;Reported server date (from 'Date' header): Wed, 08 Jan 2025 15:31:29 GMT
com.amazonaws.http.AmazonHttpClient$RequestExecutor;;executeOneRequest;;;Retrying Request: PUT https://URL /xxx/yyy/zzz.txt
com.amazonaws.http.AmazonHttpClient$RequestExecutor;;doPauseBeforeRetry;;;Retriable error detected, will retry in 891ms, attempt number: 4
....
....
....
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: null; S3 Extended Request ID: null; Proxy: xxx), S3 Extended Request ID: null
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
	at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)
	at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6532)
	at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)
	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)
	at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:169)
	at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:149)
	at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:115)
	at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:45)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	... 1 more

[kel-web]

Hi @laurit ,

We identified the root cause of the issue. By changing the Signer from S3SignerType to AWSS3V4SignerType, everything is now working as expected.

Sincerely,