Intermittent issue where S3 upload appears successful (200 OK) but object is missing

[emmymatt]

Describe the bug

We are uploading a file to S3 using a logic similar to the second example shown in the official documentation.

Here’s the relevant snippet:

try (BufferedInputStream bis = new BufferedInputStream(getInputStream())) {
    String fileName = URLEncoder.encode(this.fileName, STANDARD_CHARSET);
    PutObjectRequest putRequest = getPutRequestBuilder()
        .key(s3Key)
        .contentDisposition("filename=" + fileName)
        .build();

    BlockingInputStreamAsyncRequestBody body = AsyncRequestBody.forBlockingInputStream(null);
    UploadRequest uploadRequest = UploadRequest.builder()
        .requestBody(body)
        .putObjectRequest(putRequest)
        .build();

    Upload upload = batchDownloadConnection.s3TransferManager().upload(uploadRequest);
    body.writeInputStream(bis);

    PutObjectResponse response = upload.completionFuture().join().response();
    return response;
}

Issue:

Intermittently, we’re seeing that the uploaded file is not present in the bucket, even though the response we receive from the SDK indicates a successful upload with HTTP 200 OK. This seems to be happening only in production environment.

Here is an example of such a response:

response = PutObjectResponse(
  Expiration=expiry-date="Mon, 05 May 2025 00:00:00 GMT", rule-id="[REDACTED]",
  ETag="ec02a428cb9a3ed88ac29ea47f45820b",
  ChecksumCRC32=1HnwtQ==,
  ServerSideEncryption=AES256
),
responseMetadata = {
  Content-Length=[0],
  Date=[Fri, 04 Apr 2025 05:40:50 GMT],
  ETag=["ec02a428cb9a3ed88ac29ea47f45820b"],
  Server=[AmazonS3],
  x-amz-checksum-crc32=[1HnwtQ==],
  x-amz-checksum-type=[FULL_OBJECT],
  x-amz-expiration=[expiry-date="Mon, 05 May 2025 00:00:00 GMT", rule-id="[REDACTED]"],
  x-amz-id-2=[REDACTED],
  x-amz-request-id=[REDACTED],
  x-amz-server-side-encryption=[AES256]
},
statusText=Optional[OK],
statusCode=200
As you can see, the SDK returns a valid ETag and a 200 response. However, when we look into the bucket, the file is missing.

The AWS request ID is included above (x-amz-request-id=[REDACTED]).

Could you please investigate this behavior?

AWS SDK version: 2.27.21
AWS SDK STS version: 2.17.52

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The file should be reliably uploaded to S3, and it should be available in the bucket when the upload.completionFuture().join() call completes and returns a 200 OK response without exception.

Current Behavior

Intermittently in production, even though the SDK returns a 200 OK response with a valid ETag, the file is not present in the S3 bucket afterward.

Reproduction Steps

We have not been able to reproduce this issue consistently in non-production environments. However, the logic we use is based on the streaming upload pattern provided in the AWS documentation

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.27.21

JDK version used

17.0.7

Operating System and version

NA

[bhoradc]

Hello @emmymatt,

Thank you for reporting the issue. I understand that the behavior is intermittent in nature, and you aren't able to reproduce it reliably in a non-production environment. To investigate further, it would be helpful if you could provide the following additional information:

1. A self-contained code sample that might give us a chance to replicate the issue at our end.

2. You mentioned that you are using an older version of the AWS Java SDK (2.27.21). To ensure that the issue is not caused by a known bug in an older version, can you try upgrading to the latest stable Java SDK version and validate the behavior? The latest version is 2.31.15.

3. If you can enable verbose wire logs in your non-production setup, it would be helpful if you can capture and provide these logs when the issue happens (sporadically), which can assist in troubleshooting. Instructions to enable verbose wire logs can be found here.

4. Please provide any additional relevant information/settings (such as S3 bucket configuration) that you think might be useful in this case.

Regards,
Chaitanya

[emmymatt]

Hey @bhoradc, thanks for the response!

Below is a self-contained Java class that reproduces the issue. It stays mostly faithful to the logic we use in production where we intermittently observe the problem. Let me know if you need more details or logs.

import java.io.BufferedInputStream;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;

import software.amazon.awssdk.core.async.AsyncRequestBody;
import software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectResponse;
import software.amazon.awssdk.services.s3.model.ServerSideEncryption;
import software.amazon.awssdk.services.s3.model.Tag;
import software.amazon.awssdk.services.s3.model.Tagging;
import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.model.Upload;
import software.amazon.awssdk.transfer.s3.model.UploadRequest;

/**
 * NOTE:
 * PutObjectRequest.builder() is created once per batch of files to be processed and can be shared across threads.
 * UploadRequest.builder() and S3TransferManager.builder() would be created once per upload.
**/

public class SelfContainedS3Upload {
    public static void main(String[] args) throws Exception {

        byte[] fileData = new byte[20 * 1024];
        for (int i = 0; i < fileData.length; i++) {
            fileData[i] = 'A';
        }

        String bucketName = "BUCKET_NAME";
        String fileName = "test-file.txt";
        String s3Key = "uploads/" + fileName;

        ArrayList<Tag> objectTags = new ArrayList<>();
        objectTags.add(Tag.builder().key("retentionPeriod").value("30").build());

        PutObjectRequest.Builder requestBuilder = PutObjectRequest.builder()
            .bucket(bucketName)
            .serverSideEncryption(ServerSideEncryption.AES256)
            .tagging(Tagging.builder().tagSet(objectTags).build());

        S3TransferManager transferManager = S3TransferManager.builder().build();
        InputStream inputStream = new ByteArrayInputStream(fileData); // Actually a stream from FTP endpoint

        try (BufferedInputStream bis = new BufferedInputStream(inputStream)) {
            String encodedFileName = URLEncoder.encode(fileName, StandardCharsets.UTF_8);
            PutObjectRequest putRequest = requestBuilder.key(s3Key).contentDisposition("filename=" + encodedFileName).build();
            BlockingInputStreamAsyncRequestBody body = AsyncRequestBody.forBlockingInputStream(null);
            UploadRequest uploadRequest = UploadRequest.builder().requestBody(body).putObjectRequest(putRequest).build();
            Upload upload = transferManager.upload(uploadRequest);
            body.writeInputStream(bis);
            PutObjectResponse response = upload.completionFuture().join().response();
            System.out.println("Upload response: " + response);
        }

            S3Client s3Client = S3Client.builder().build();
            HeadObjectRequest headRequest = HeadObjectRequest.builder()
                .bucket(bucketName)
                .key(s3Key)
                .build();

            HeadObjectResponse headResponse = s3Client.headObject(headRequest);
            System.out.println("HeadObject confirms upload. Size: " + headResponse.contentLength() + " bytes, ETag: " + headResponse.eTag());
    }
}

I'll try testing with the latest AWS SDK version in production and share any findings by Monday EOD.

Regarding wire logging: since this issue only occurs in production and could expose sensitive content (e.g., file data), we’re hesitant to enable full Netty logging. I looked into this approach for masking sensitive logs using Log4j2, and we’ll evaluate if it gives us enough control to consider enabling netty logs safely.

Regarding the S3 bucket configuration — I’m afraid I don’t have full context on that at the moment. Could you let me know specifically what details you’re looking for and I’ll get back to you with the relevant information?

In terms of traffic, we’re seeing upwards of 200,000 uploads per day. The issue affects a small fraction — roughly 0.1% of uploads — but it’s concerning because the response from the SDK returns 200 OK with a valid ETag, so we have no indication anything went wrong until we attempt to fetch the object again and find it missing.

Also, were you able to gather any insights from the AWS request ID I had shared earlier? Let me know if it helped in tracing anything on your end.

[Fred1155]

Hi could you also please reach out to S3 customer support for this issue? You can do that by creating a technical support request. We can have the follow-up discussion there.

[emmymatt]

Got it @Fred1155. Just to clarify, I'll need to open a support case at https://support.console.aws.amazon.com/support/home, right?

[Fred1155]

Yes

[emmymatt]

We raised an S3 support request and confirmed that the earlier PUT request is writing to a different S3 key - one that was actually intended for a different file being processed in another thread.

In our current implementation, we construct a single shared PutObjectRequest.Builder like this:

PutObjectRequest.Builder requestBuilder = PutObjectRequest.builder()
    .bucket(bucketName)
    .serverSideEncryption(ServerSideEncryption.AES256)
    .tagging(Tagging.builder().tagSet(objectTags).build());

We then reuse this builder across multiple threads to construct PutObjectRequests for different files in a batch. We now suspect this shared usage across threads is causing key collisions.

To address this, we're planning to construct a fresh PutObjectRequest per file like this:

PutObjectRequest putRequest = PutObjectRequest.builder()
    .bucket(bucketName)
    .serverSideEncryption(ServerSideEncryption.AES256)
    .tagging(Tagging.builder().tagSet(objectTags).build())
    .key(s3Key)
    .contentDisposition("filename=" + encodedFileName)
    .build();

Could you please confirm if our understanding is correct based on the given details, and whether this is the recommended approach for thread-safe usage of request builders?

Thanks!

[Fred1155]

Your understanding is correct. The builder itself is not immutable when the .build() is not called. A safer way is to construct a fresh PutObjectRequest per file and that is thread-safe.

[emmymatt]

okay @Fred1155. I will go ahead and close the issue.
Thank you for the support!