Re Opening Race conditions in S3 upload #6115

[radding]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

This reopens #6115

I have encountered this issue using the Open Next CDK constructs. I have also found a potential work around for this issue in case it is useful to find a root cause.

The code that causes this issue is this code here: https://github.com/jetbridge/cdk-nextjs/blob/d5c116b0bffb14cc28928c32362946e9aef807b1/src/lambdas/nextjs-bucket-deployment.ts#L296

The solution to work around the bug is: https://github.com/radding/cdk-nextjs/blob/main/src/lambdas/nextjs-bucket-deployment.ts#L299C4-L311C36

As you can see in those two code samples, the work around is moving from uploading content in parallel to uploading content in serial. The await putObjectInputs.reduce(async (acc, params) waits for the previous upload to complete before attempting to upload the next object.

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/lib-storage@3.758.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.11.0

Reproduction Steps

1. Have a directory of files of various sizes, some images, some HTML, some Javascript
2. Execute this code:

const fs = require('fs');

Promise.all(fs.readdirSync(testFolder).map(file => {
  const upload = new Upload({.., params: { Key: file, Body: fs.readFileSync(file), Bucket: <target_bucket> })
  return upload.done();
}));

Observed Behavior

Results in the following error and no upload: XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed.

Expected Behavior

No error and upload is successful.

Possible Solution

My hypothesis is that there is something sharing memory between upload requests, as switch to serial uploads solves the problem. I suspect the problem is with in the multipart upload code, but I have not had the opportunity to dig in any deeper than this.

Additional Information/Context

This is basically just adding on to #6115

[zshzbh]

Hey @radding ,

I can't reproduce this issue.
Code I have -

const s3 = new S3Client({ region: "us-east-1" });

const bucket = "new-bucket-maggie-ma";
const filePath = "./";

Promise.all(
  fs.readdirSync(filePath).map((file) => {
    const upload = new Upload({
      client: s3,
      params: {
        Bucket: bucket,
        Key: file,
        Body: fs.readFileSync(file),
      },

    });
    console.log(`Uploading ${file}`);
    return upload.done();
  })
);

I was able to upload 95 files, 20 of them are of 2.6 MB each. In order to dive deep into this issue, could you please provide-

• how many files you are uploading at the same time?
• What are the sizes of the files you were trying to upload?

Thanks!

Updates- I also added 10 files, each of them is ~23MB, and uploading 105 files still works.

[radding]

Since my project is public, I can just give you the same directory that was giving me trouble. I have attached it. It is a nextJS build artifact.

If you don't want to use those assets, I am uploading ~311 files running in a NodeJS lambda (not sure if that makes a difference). Here is the configuration for that lambda:

    architecture: Architecture.ARM_64,
    memorySize: 1536,
    runtime: Runtime.NODEJS_20_X,

assets.zip

[bestickley]

@zshzbh, were you able to reproduce above? I have a user of library I maintain that is experiencing similar issue in cdklabs/cdk-nextjs#105. As temporary workaround I'm going to do something similar to the PR: jetbridge/cdk-nextjs#249 but I don't think user's of @aws-sdk/lib-storage should have to worry about this. This should be fixed in library.

If the issue is resolved by user's awaiting in serial rather than in parallel and the error that is experienced is XAmzContentSHA256Mismatch then I'd guess that the library is maybe sending the wrong sha256 hash for requests. Both of these issues are in the domain of uploading Next.js static assets to S3. Static assets can be images that have large files sizes. In your tests, have you tried large file sizes?

[zshzbh]

No I was no able to reproduce it. Could you please give me an example on how large is the file?

[github-actions]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.