put_object throwing Access Denied for IBM COS buckets

[rajeev36]

Describe the bug

put_object is throwing Access Denied error with version 1.11.518
But with the same credentials put_object is working with version 1.9.359

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

put_object should work for IBM COS

Current Behavior

put_object throwing error Access Denied

Reproduction Steps

tried with sample put_object program

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

1.11.518

Compiler and Version used

c++11 and gcc9

Operating System and version

RHEL 8.10

[sbiscigl]

put_object should work for IBM COS

We do not have any control over IBM COS. I would suggest reaching out to IBM for support. If you see errors with AWS S3 we can help.

[rajeev36]

Thank you, @sbiscigl , for your reply.

I'm trying to run the sample program below using SDK versions 1.9.359 and 1.11.549 (with the same credentials and bucket).

The program executed successfully with version 1.9.359,
but it failed with version 1.11.549, throwing the following exception:

Exception name: AccessDenied
Error message: Access Denied

Sample program

#include <aws/core/Aws.h>
#include <aws/core/utils/logging/ConsoleLogSystem.h>
#include <aws/core/utils/logging/LogLevel.h>
#include <aws/core/client/ClientConfiguration.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/PutObjectRequest.h>
#include <iostream>

#include <aws/core/auth/AWSCredentials.h>
#include <aws/core/auth/AWSCredentialsProvider.h>

int main() {
    Aws::SDKOptions options;

    // Enable logging
    options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Debug;
    options.loggingOptions.logger_create_fn = [] {
        return std::make_shared<Aws::Utils::Logging::ConsoleLogSystem>(
            Aws::Utils::Logging::LogLevel::Debug);
    };

    Aws::InitAPI(options);
    {
        const Aws::String bucket_name = "XXX";
        const Aws::String object_key = "/tmp/rajeev.txt";
        const Aws::String object_data = "Hello from AWS SDK  with custom endpoint!";
        const Aws::String endpoint = "XXXXXX"; // e.g., MinIO

        Aws::Client::ClientConfiguration config;
        config.endpointOverride = endpoint;
        config.scheme = Aws::Http::Scheme::HTTP;
        config.region = "XXX";
        config.verifySSL = false;
        Aws::Auth::AWSCredentials creds("XXXX", "XXX");


        // You pass `false` as the third param to the S3Client constructor to disable virtual addressing
        Aws::S3::S3Client s3_client(config,
                                    Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never,
                                    false); // virtual_hosted_style = false

        auto input_data = Aws::MakeShared<Aws::StringStream>("PutObjectTag");
        *input_data << object_data;

        Aws::S3::Model::PutObjectRequest request;
        request.SetBucket(bucket_name);
        request.SetKey(object_key);
        request.SetBody(input_data);

        auto outcome = s3_client.PutObject(request);

        if (outcome.IsSuccess()) {
            std::cout << "✅ Successfully uploaded object to custom S3 endpoint." << std::endl;
        } else {
            std::cerr << "❌ Failed to upload object: "
                      << outcome.GetError().GetExceptionName() << " - "
                      << outcome.GetError().GetMessage() << std::endl;
        }
    }
    Aws::ShutdownAPI(options);
    return 0;
}

I can see the config level difference while comparing the debug logs

Attaching the debug logs with both the versions for reference

v1.11.549.log

v1_9_359.log

Can you please help on this.

[SergeyRyabinin]

Hello,
thank you for the log files provided,
it seems that SDK fails to parse your aws configuration file:

Aws::Config::ConfigFileProfileFSM [140381300998656] Unknown property: aws_access_key_id in the profile: localstack
Aws::Config::ConfigFileProfileFSM [140381300998656] Unknown property: aws_secret_access_key in the profile: localstack
Aws::Config::ConfigFileProfileFSM [140381300998656] found access key YYYYYYYY
Aws::Config::ConfigFileProfileFSM [140381300998656] Unknown property: aws_access_key_id in the profile: default
Aws::Config::ConfigFileProfileFSM [140381300998656] Unknown property: aws_secret_access_key in the profile: default
Aws::Config::ConfigFileProfileFSM [140381300998656] found access key XXXXXXXX
Aws::Config::AWSProfileConfigLoaderBase [140381300998656] Successfully reloaded configuration.
FileSystemUtils [140381300998656] Environment value for variable HOME is /home/nz
FileSystemUtils [140381300998656] Home directory is missing the final / appending one to normalize
FileSystemUtils [140381300998656] Final Home Directory is /home/nz/
Aws::Config::ConfigFileProfileFSM [140381300998656] Found region us-east-2
Aws::Config::ConfigFileProfileFSM [140381300998656] In configuration files, the profile name must start with profile keyword (except default profile): [services s3-specific]
Aws::Config::ConfigFileProfileFSM [140381300998656] Found region ap-geo

The SDK does not support yet services sections introduced on the CLI and other SDKs, but it seems that the credentials are failed to be fetched either from the profile sections.

Could you please provide an example how your profile and config files look?
PLEASE DO SANITIZE THEM AND REMOVE ANY CONFIDENTIAL INFO, we need only a high-level example of the how they are structured.

Thank you.

[rajeev36]

Hello @SergeyRyabinin

We do not maintain any profile or configuration files to store the credentials. Instead, we are passing the credentials directly within the program, as shown below.

        Aws::Auth::AWSCredentials creds("XXXX", "XXX");

        // You pass `false` as the third param to the S3Client constructor to disable virtual addressing
        Aws::S3::S3Client s3_client(config,
                                    Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never,
                                    false); // virtual_hosted_style = false

Our concern is why the same program, using the same IBM COS credentials, works with SDK version 1.9.359, but not with version 1.11.x.

Attached are the test results of the put-obj API (version 1.11) using S3 credentials (successful) and IBM COS credentials (errored).

cos_creds.log
s3_creds.log

Could you please help us understand if any configuration-level changes are needed to make the program work with SDK version 1.11.X?

[SergeyRyabinin]

Hi,

We do not maintain any profile or configuration files to store the credentials.

According to the log file provided, you have some non-empty config file ~/.aws/config on your system. It maybe generated by the AWS CLI.
However, for the most recent log provided, I can confirm that you are providing credentials in code.

Could you please help us understand if any configuration-level changes are needed to make the program work with SDK version 1.11.X?

I've checked your most recent logs and the main difference in the HTTP request sent I see is

new 1.11.549:

PUT /path/to/object.txt HTTP/1.1
Host: s3.XXXX.YYY
Accept: */*
amz-sdk-invocation-id: DEADBEEF-BAAD-F00D-CAFE-EEEEEEEEEEEE
amz-sdk-request: attempt=1
authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXX/17890418/bl-bla/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-trailer, Signature=SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSIGNATURE
content-encoding: aws-chunked
content-type: binary/octet-stream
transfer-encoding: chunked
user-agent: aws-sdk-cpp/1.11.549 ua/2.1 api/S3 os/Linux#4.18.0... lang/c++#C++11 md/aws-crt#0.32.4-dev+0505f3fd md/arch#x86_64 md/GCC#9.2.0 m/D,W,Z,b
x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
x-amz-date: 17890418T151236Z
x-amz-decoded-content-length: 46
x-amz-trailer: x-amz-checksum-crc64nvme
Expect: 100-continue

while the old 1.9.x is

PUT /path/to/object.txt HTTP/1.1
Host: s3.XXXX.YYY
Accept: */*
amz-sdk-invocation-id: DEADBEEF-BAAD-F00D-CAFE-EEEEEEEEEEEE
amz-sdk-request: attempt=1
authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXX/17890418/bl-bla/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSIGNATURE
content-length: 46
content-md5: bmljZSB0byBzZWUgeW91IQ==
content-type: binary/octet-stream
user-agent: aws-sdk-cpp/1.9.359 Linux/4.18.0.....
x-amz-content-sha256: some_hash
x-amz-date: 17890418T151253Z
Expect: 100-continue

So the main difference is presence of content-md5: bmljZSB0byBzZWUgeW91IQ== and content-encoding: aws-chunked header. And a different value for the x-amz-content-sha256 header.

It comes from a new default integrity feature of AWS S3.
Please see the announcement: #3253
and this documentation page about how to disable it: https://github.com/aws/aws-sdk-cpp/blob/main/docs/MD5ChecksumFallback.md

Thanks.

[rajeev36]

Thank you @SergeyRyabinin .

its working now with config setting

clientConfig.checksumConfig.requestChecksumCalculation = Aws::Client::RequestChecksumCalculation::WHEN_REQUIRED;

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.