XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed

[sumitjhajharia]

Checkboxes for prior research

• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.
• I've gone through Developer Guide and API reference

Describe the bug

Hi,

We are getting XAmzContentSHA256Mismatch issue randomly when try to upload ~400 files to S3.
From primary investigation its looks like it fails when there is race condition OR multi thread condition.

Checked similar mentioned issues - aws/aws-sdk-ruby#1594 , #6115 , but it seems it's still not fixed.

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

"@aws-sdk/client-s3": "3.744.0", "@aws-sdk/lib-storage": "3.744.0"

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

20.14.0

Reproduction Steps

const s3Client = new S3Client({
  credentials: { accessKeyId, secretAccessKey },
  region,
  signatureVersion: 'v4',
  maxRetries: 5,
});

const uploadFileToS3 = ({client, key, file, uploadParams}) => {
    const body = fs.createReadStream(file);
  const command = new Upload({
    client,
    params: {
      Key: key,
      Body: body,
      Bucket: bucket,
      BucketName: bucketName,
    },
    leavePartsOnError: false, // default - abort the multipart upload if an error occurs, automatically call AbortMultipartUpload
    queueSize: 4, // default -  queue manager to upload parts in parallel
  });
  const promise = command
    .done()
    .then((response) => response)
    .catch((error) => {
      throw new Error(error);
    });
  return { promise };
};

and calling uploadFileToS3 in Promise.all([<uploadFileToS3(...), uploadFileToS3(...), uploadFileToS3(...)>]);

Observed Behavior

` XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed.

#43 281.4     at throwDefaultError (/app/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)
#43 281.4     at /app/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5
#43 281.4     at de_CommandError (/app/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4970:14)
#43 281.4     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
#43 281.4     at async /app/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
#43 281.4     at async /app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:483:18
#43 281.4     at async /app/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
#43 281.4     at async /app/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:315:18
#43 281.4     at async /app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:109:22
#43 281.4     at async /app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:136:14 {
#43 281.4   '$fault': 'client',
#43 281.4   '$metadata': {
#43 281.4     httpStatusCode: 400,
#43 281.4     requestId: '5PD542KYBKS97QDX',
#43 281.4     extendedRequestId: '+o3EAtt2QMJCuG9zqoqlbmWAA1bZ9gn8vRuQBA8J20XLVQoWzwOBsDL31Cp9fK5DNKfuq56/GvUSoN0ro6qIPg==',
#43 281.4     cfId: undefined,
#43 281.4     attempts: 1,
#43 281.4     totalRetryDelay: 0
#43 281.4   },

#43 281.4   Code: 'XAmzContentSHA256Mismatch',
#43 281.4   ClientComputedContentSHA256: 'f5df4223ba30e7d19fd37ee4c61628b9712f91998b320473abc51d64bc04da83',
#43 281.4   S3ComputedContentSHA256: '626a70019720d2d30bd88df3fccd18fa723a7dfa69412bffe62c0542f5abd266',
#43 281.4   RequestId: '5PD542KYBKS97QDX',
#43 281.4   HostId: '+o3EAtt2QMJCuG9zqoqlbmWAA1bZ9gn8vRuQBA8J20XLVQoWzwOBsDL31Cp9fK5DNKfuq56/GvUSoN0ro6qIPg=='
#43 281.4 }
#43 281.4 Failed to compile.
#43 281.4 
#43 281.4 XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was `computed.``

Expected Behavior

aws sdk should create and match authorization headers if not provided manually.

Possible Solution

No response

Additional Information/Context

No response

[zshzbh]

Hey @sumitjhajharia ,

I can't reproduce this issue using the version "@aws-sdk/client-s3": "3.744.0", "@aws-sdk/lib-storage": "3.744.0".

As you mentioned that you are uploading ~400 files, I assume that is related to the amount of the files. We found that reducing the batch size to 5 seems to resolve the issue. While I'm still waiting for service team to get an update, could you please lower the batch size as a workaround?

[zshzbh]

Posting my code -

import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import fs from "fs"; 

const s3Client = new S3Client({
  region: "us-east-1",
  signatureVersion: "v4",
  maxRetries: 5,
});

async function uploadFileToS3({ client, key, file }) {
  try {
    const fileContent = await fs.readFile(file);
    const command = new Upload({
      client,
      params: {
        Bucket: "new-xxx-xxx-ma",
        Key: key,
        Body: fileContent,
      },
      leavePartsOnError: false,
      queueSize: 4,
    });

    const response = await command.done();
    return response;
  } catch (error) {
    throw new Error(`Failed to upload file: ${error.message}`);
  }
}


async function main() {
  try {
    const result = await uploadFileToS3({
      client: s3Client,
      key: "6893.txt",
      file: "test.txt"
    });
    console.log("Upload successful:", result);
  } catch (error) {
    console.error("Upload failed:", error);
  }
}

main();

[zshzbh]

Hey @sumitjhajharia ,

I want to check if you are seeing this issue in lambda?

[sumitjhajharia]

No @zshzbh , it's not in lambda.

In above shared code snippet can you try uploading multiple files using Promise.all(<>) and use streams instead of readFile.

Its not failing for single fail or low no of files(20-30), it fails randomly when uploading >50 files.

[zshzbh]

Thanks for the update! I will try uploading >50 files and see if this still happens.

[zshzbh]

Hey @sumitjhajharia ,

What is the implementation to upload multiple files using uploadFileToS3? Do you call uploadFileToS3 for each file OR you put uploading multiple files logic within uploadFileToS3?

I implemented multi file S3 upload functionality by creating an array of promises that handle 80 separate file uploads. Each promise executes the uploadFileToS3 function for its respective file, and all these upload operations are consolidated in the uploadPromises . And the result is const results = await Promise.all(uploadPromises); . I'm still not able to reproduce this issue. I'm uploading 80 files, each file is ~2.6 MB. I run the code couple times. This is the code I have -

import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import fs from "fs";
import { createReadStream } from "fs";

const s3Client = new S3Client({
  region: "us-east-1",
  signatureVersion: "v4",
  maxRetries: 5,
});

async function uploadFileToS3({ client, key, file }) {
  try {
    const fileStream = createReadStream(file);

    const command = new Upload({
      client,
      params: {
        Bucket: "new-bucket-maggie-ma",
        Key: key,
        Body: fileStream,
      },
      leavePartsOnError: false,
      queueSize: 4,
    });

    //  Progress tracking
    command.on("httpUploadProgress", (progress) => {
      console.log(`Progress for ${key}: ${progress.loaded}/${progress.total}`);
    });

    const response = await command.done();
    return response;
  } catch (error) {
    throw new Error(`Failed to upload file: ${error.message}`);
  }
}

async function uploadMultipleFiles(files) {
  const uploadPromises = files.map(({ key, filePath }) => {
    return uploadFileToS3({
      client: s3Client,
      key,
      file: filePath,
    })
      .then(() => ({
        key,
        message: `Successfully uploaded ${key}`,
      }))
      .catch((error) => ({
        key,
        message: `Failed to upload ${key}: ${error.message}`,
      }));
  });

  try {
    const results = await Promise.all(uploadPromises);
    return results;
  } catch (error) {
    console.error(" Upload failed:", error);
    throw error;
  }
}

async function main() {
  try {
    // Example files to upload, upload 80 files
    const filesToUpload = Array.from({ length: 80 }, (_, index) => ({
      key: `file${index + 1}.txt`,
      filePath: "./test.txt", // each txt file is 2.6 mb
    }));

    console.log("Uploading 80 files ");
    const results = await uploadMultipleFiles(filesToUpload);
  } catch (error) {
    console.error("Main process failed:", error);
  }
}

main();

And this is the result I got

➜  6893 node index.mjs
Uploading 80 files 
Progress for file9.txt: 2630972/2630972
Progress for file35.txt: 2630972/2630972
Progress for file31.txt: 2630972/2630972
Progress for file4.txt: 2630972/2630972
Progress for file10.txt: 2630972/2630972
Progress for file75.txt: 2630972/2630972
Progress for file12.txt: 2630972/2630972
Progress for file3.txt: 2630972/2630972
Progress for file46.txt: 2630972/2630972
Progress for file64.txt: 2630972/2630972
Progress for file24.txt: 2630972/2630972
Progress for file5.txt: 2630972/2630972
Progress for file17.txt: 2630972/2630972
Progress for file7.txt: 2630972/2630972
Progress for file42.txt: 2630972/2630972
Progress for file11.txt: 2630972/2630972
Progress for file20.txt: 2630972/2630972
Progress for file6.txt: 2630972/2630972
Progress for file14.txt: 2630972/2630972
Progress for file40.txt: 2630972/2630972
Progress for file41.txt: 2630972/2630972
Progress for file26.txt: 2630972/2630972
Progress for file21.txt: 2630972/2630972
Progress for file59.txt: 2630972/2630972
Progress for file47.txt: 2630972/2630972
Progress for file19.txt: 2630972/2630972
Progress for file38.txt: 2630972/2630972
Progress for file18.txt: 2630972/2630972
Progress for file27.txt: 2630972/2630972
Progress for file51.txt: 2630972/2630972
Progress for file37.txt: 2630972/2630972
Progress for file43.txt: 2630972/2630972
Progress for file55.txt: 2630972/2630972
Progress for file22.txt: 2630972/2630972
Progress for file25.txt: 2630972/2630972
Progress for file8.txt: 2630972/2630972
Progress for file23.txt: 2630972/2630972
Progress for file36.txt: 2630972/2630972
Progress for file33.txt: 2630972/2630972
Progress for file44.txt: 2630972/2630972
Progress for file50.txt: 2630972/2630972
Progress for file16.txt: 2630972/2630972
Progress for file30.txt: 2630972/2630972
Progress for file32.txt: 2630972/2630972
Progress for file58.txt: 2630972/2630972
Progress for file15.txt: 2630972/2630972
Progress for file52.txt: 2630972/2630972
Progress for file13.txt: 2630972/2630972
Progress for file2.txt: 2630972/2630972
Progress for file1.txt: 2630972/2630972
Progress for file28.txt: 2630972/2630972
Progress for file61.txt: 2630972/2630972
Progress for file62.txt: 2630972/2630972
Progress for file66.txt: 2630972/2630972
Progress for file48.txt: 2630972/2630972
Progress for file54.txt: 2630972/2630972
Progress for file39.txt: 2630972/2630972
Progress for file76.txt: 2630972/2630972
Progress for file57.txt: 2630972/2630972
Progress for file69.txt: 2630972/2630972
Progress for file68.txt: 2630972/2630972
Progress for file77.txt: 2630972/2630972
Progress for file70.txt: 2630972/2630972
Progress for file80.txt: 2630972/2630972
Progress for file72.txt: 2630972/2630972
Progress for file78.txt: 2630972/2630972
Progress for file67.txt: 2630972/2630972
Progress for file71.txt: 2630972/2630972
Progress for file49.txt: 2630972/2630972
Progress for file56.txt: 2630972/2630972
Progress for file73.txt: 2630972/2630972
Progress for file29.txt: 2630972/2630972
Progress for file53.txt: 2630972/2630972
Progress for file63.txt: 2630972/2630972
Progress for file34.txt: 2630972/2630972
Progress for file74.txt: 2630972/2630972
Progress for file65.txt: 2630972/2630972
Progress for file45.txt: 2630972/2630972
Progress for file60.txt: 2630972/2630972
Progress for file79.txt: 2630972/2630972

I don't know what is the code you use to upload multiple files, could you please try the code I posted and see if it's working? If the code I provided is still not working on your end, then I think it could be related to the service. And we can further investigate that