New checksum header seems to break presigned URLs

[atwam]

Describe the bug

Issue

When using the presign functionality, a new x-amx-checksum-mode header gets added to the request after its signature, causing validation of presigned URL to fail.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

New checksum header gets added before signing, so the signed url is correct.

Current Behavior

When using the presign functionality, a new x-amx-checksum-mode header gets added to the request after its signature, causing validation to fail with:

<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<Error><Code>AccessDenied</Code>
<Message>There were headers present in the request which were not signed</Message>
<HeadersNotSigned>x-amz-checksum-mode</HeadersNotSigned>
<RequestId>...</RequestId>
<HostId>...</HostId>
</Error>"

Reproduction Steps

I have not extracted a fully running code, can do if necessary, but the gist of it is the following:

let presigning_config = PresigningConfig::builder()
     .expires_in(get_expiry())
     .build()
     .unwrap();

let request = self.client
            .get_object()
            .bucket(bucket_name)
            .key(key)
            .request_payer(RequestPayer::Requester)
            .presigned(presigning_config)
            .await?;

// request.headers() has `x-amz-request-payer` and `x-amz-checksum-mode` headers

Trying to access the signed URL now fails with the error above, which seems to imply that its signature happens before addition of the x-amz-checksum-mode header.

I have looked at #1246 , which also seems to be related to a late addition of the header.

Possible Solution

I think the HttpResponseChecksumDecorator runs after the PreSigningInterceptor.

Short term solution would be to ensure that PreSigningInterceptor runs last, or alternatively have it somehow "freeze" the headers/payload, making it read-only, so that future similar regressions don't happen.

A short term workaround in our case is to set the checksum config to WhenRequired to prevent decoration with the new header. Another solution is to manually remove the newly-added header from the presigned request (at the risk of having again verification failure once the header is properly included in the signature). But this is a local hack-ish workaround, not a proper fix of the s3 sdk.

Version

Versions used

├── aws-config v1.5.14
│   ├── aws-credential-types v1.2.1
│   │   ├── aws-smithy-async v1.2.4
│   │   ├── aws-smithy-runtime-api v1.7.3
│   │   │   ├── aws-smithy-async v1.2.4 (*)
│   │   │   ├── aws-smithy-types v1.2.12
│   │   ├── aws-smithy-types v1.2.12 (*)
│   ├── aws-runtime v1.5.4
│   │   ├── aws-credential-types v1.2.1 (*)
│   │   ├── aws-sigv4 v1.2.7
│   │   │   ├── aws-credential-types v1.2.1 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.6
│   │   │   │   ├── aws-smithy-types v1.2.12 (*)
│   │   │   ├── aws-smithy-http v0.60.12
│   │   │   │   ├── aws-smithy-eventstream v0.60.6 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   │   │   ├── aws-smithy-types v1.2.12 (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   │   ├── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-smithy-async v1.2.4 (*)
│   │   ├── aws-smithy-eventstream v0.60.6 (*)
│   │   ├── aws-smithy-http v0.60.12 (*)
│   │   ├── aws-smithy-runtime v1.7.7
│   │   │   ├── aws-smithy-async v1.2.4 (*)
│   │   │   ├── aws-smithy-http v0.60.12 (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   │   ├── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   ├── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-types v1.3.4
│   │   │   ├── aws-credential-types v1.2.1 (*)
│   │   │   ├── aws-smithy-async v1.2.4 (*)
│   │   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   │   ├── aws-smithy-types v1.2.12 (*)
│   ├── aws-sdk-sso v1.55.0
│   │   ├── aws-credential-types v1.2.1 (*)
│   │   ├── aws-runtime v1.5.4 (*)
│   │   ├── aws-smithy-async v1.2.4 (*)
│   │   ├── aws-smithy-http v0.60.12 (*)
│   │   ├── aws-smithy-json v0.61.2
│   │   │   └── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-smithy-runtime v1.7.7 (*)
│   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   ├── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-types v1.3.4 (*)
│   ├── aws-sdk-ssooidc v1.56.0
│   │   ├── aws-credential-types v1.2.1 (*)
│   │   ├── aws-runtime v1.5.4 (*)
│   │   ├── aws-smithy-async v1.2.4 (*)
│   │   ├── aws-smithy-http v0.60.12 (*)
│   │   ├── aws-smithy-json v0.61.2 (*)
│   │   ├── aws-smithy-runtime v1.7.7 (*)
│   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   ├── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-types v1.3.4 (*)
│   ├── aws-sdk-sts v1.56.0
│   │   ├── aws-credential-types v1.2.1 (*)
│   │   ├── aws-runtime v1.5.4 (*)
│   │   ├── aws-smithy-async v1.2.4 (*)
│   │   ├── aws-smithy-http v0.60.12 (*)
│   │   ├── aws-smithy-json v0.61.2 (*)
│   │   ├── aws-smithy-query v0.60.7
│   │   │   ├── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-smithy-runtime v1.7.7 (*)
│   │   ├── aws-smithy-runtime-api v1.7.3 (*)
│   │   ├── aws-smithy-types v1.2.12 (*)
│   │   ├── aws-smithy-xml v0.60.9
│   │   ├── aws-types v1.3.4 (*)
│   ├── aws-smithy-async v1.2.4 (*)
│   ├── aws-smithy-http v0.60.12 (*)
│   ├── aws-smithy-json v0.61.2 (*)
│   ├── aws-smithy-runtime v1.7.7 (*)
│   ├── aws-smithy-runtime-api v1.7.3 (*)
│   ├── aws-smithy-types v1.2.12 (*)
│   ├── aws-types v1.3.4 (*)
├── aws-credential-types v1.2.1 (*)
├── aws-sdk-s3 v1.70.0
│   ├── aws-credential-types v1.2.1 (*)
│   ├── aws-runtime v1.5.4 (*)
│   ├── aws-sigv4 v1.2.7 (*)
│   ├── aws-smithy-async v1.2.4 (*)
│   ├── aws-smithy-checksums v0.62.0
│   │   ├── aws-smithy-http v0.60.12 (*)
│   │   ├── aws-smithy-types v1.2.12 (*)
│   ├── aws-smithy-eventstream v0.60.6 (*)
│   ├── aws-smithy-http v0.60.12 (*)
│   ├── aws-smithy-json v0.61.2 (*)
│   ├── aws-smithy-runtime v1.7.7 (*)
│   ├── aws-smithy-runtime-api v1.7.3 (*)
│   ├── aws-smithy-types v1.2.12 (*)
│   ├── aws-smithy-xml v0.60.9 (*)
│   ├── aws-types v1.3.4 (*)
├── aws-smithy-runtime-api v1.7.3 (*)
├── aws-smithy-types v1.2.12 (*)
├── aws-types v1.3.4 (*)

Environment details (OS name and version, etc.)

Linux 6.8.0-51-generic-Ubuntu SMP PREEMPT_DYNAMIC x86_64 x86_64 x86_64 GNU/Linux

[AmyJoT]

Also experiencing this issue here. Looks like it started 2 weeks ago

[joeroback]

i am also running into the same issue

[AmyJoT]

Got a workaround here by adding the following RequestCheckSumCalculation to the client - it was previously working without this, and we didn't upgrade anything. This only occurred in our deployed app. In local testing the headers did not appear.

For local testing, setting a checksum mode on the presigned builder revealed this would be a workaround.

       use aws_sdk_s3::config::RequestChecksumCalculation;

        let shared_config = aws_config::load_defaults(aws_config::BehaviorVersion::latest()).await;

        let s3_config: Config = Builder::from(&shared_config)
            .request_checksum_calculation(RequestChecksumCalculation::WhenRequired)
            .build();

        let inner = aws_sdk_s3::Client::from_conf(s3_config);

[landonxjames]

Could you share some details about how you are calling the presigned request? Are you making it against S3 or another service with an S3-like API?

We have an example of this in our CI that continues to work and I was just able to successfully run both presigned GET and PUT requests to my own bucket with the following dependencies in my Cargo.toml:

[dependencies]
aws-config = "1.5.16"
aws-sdk-s3 = "1.73.0"

For reference my test code looks like:

let presigning_config = PresigningConfig::builder()
        .expires_in(std::time::Duration::from_secs(300))
        .build()
        .unwrap();

    let get_request = client
        .get_object()
        .bucket("NOT_A_REAL_BUCKET")
        .key("NOT_A_REAL_KEY")
        .presigned(presigning_config)
        .await
        .unwrap();

    let resp = reqwest::get(get_request.uri().to_string())
        .await
        .unwrap()
        .text()
        .await
        .unwrap();

    println!("PRESIGNED GET RESPONSE: {resp:#?}")

[atwam]

We are making our request directly to s3.
Running a test similar to yours, I think the issue is that your reqwest::get does not use the headers returned in the PresignedRequest.

Looking at it on our side, and just printing the URL, I get:

let client = S3Client::from_env().build().await;
let presigning_config = PresigningConfig::builder()
            .expires_in(std::time::Duration::from_secs(300))
            .build()
            .unwrap();

let request = self
            .get_object()
            .bucket("PRIVATE_BUCKET")
            .key("PRIVATE_KEY")
            .request_payer(RequestPayer::Requester)
            .presigned(presigning_config)
            .await
            .unwrap();

println!("Presigned URI: {}", presigned.uri());
println!("Presigned headers: {:?}", presigned.headers().collect::<Vec<_>>());

Presigned URI: https://PRIVATE_BUCKET.s3.us-east-1.amazonaws.com/PRIVATE_KEY?x-id=GetObject&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ABCDE%2F20250205%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20250205T090127Z&X-Amz-Expires=54&X-Amz-SignedHeaders=host%3Bx-amz-request-payer&X-Amz-Signature=1234567890&X-Amz-Security-Token=IQo
Presigned headers: [("x-amz-request-payer", "requester"), ("x-amz-checksum-mode", "ENABLED")]

My guess is that your test works because it is just using the normal URI for its GET request, and not adding any header.

In our case, you can see that the presigned request has headers x-amz-request-payer and x-amz-checksum-mode.
But the X-Amz-SignedHeaders query param in the presigned URI only includes x-amz-request-payer, and not x-amz-checksum-mode.

Changing your test to have a header before presigning (such as request_payer(RequestPayer::Requester)), and making
sure you use these in your GET request by using:

let resp = reqwest::get(get_request.uri().to_string()).headers(get_request.headers())

will use whatever headers are included in the presigned url, and should fail like ours do. It might be worth expanding your test slightly, to include headers before presigning happens.

[landonxjames]

Thanks for the info, was able to repro with that and am working on a fix

[landonxjames]

Fix for this was merged in smithy-lang/smithy-rs#4000 and should deploy to crates.io early next week

[ysaito1001]

The fix was released as part of release-2025-02-13. Let us know if you still observe the issue after you've upgraded aws-sdk-s3 to 1.75.0.