MissingContentMD5 error on `DeleteObjects` command

[aseemk]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

Attempting to delete multiple objects in a bucket, using the DeleteObjects command like this:

await this.client.send(
  new DeleteObjectsCommand({
    Bucket: this.bucket,
    Delete: { Objects: keys.map(key => ({ Key: key })) },
  }),
);

Fails at runtime with this error:

{
  "type": "S3ServiceException",
  "message": "Missing required header for this request: Content-Md5.",
  "stack":
      MissingContentMD5: Missing required header for this request: Content-Md5.
          at throwDefaultError (<my project path>/node_modules/.pnpm/@smithy+smithy-client@4.1.6/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)
          at <my project path>/node_modules/.pnpm/@smithy+smithy-client@4.1.6/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5
          at de_CommandError (<my project path>/node_modules/.pnpm/@aws-sdk+client-s3@3.758.0/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4971:14)
          at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
          at async <my project path>/node_modules/.pnpm/@smithy+middleware-serde@4.0.2/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
          at async <my project path>/node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.758.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:484:18
          at async <my project path>/node_modules/.pnpm/@smithy+middleware-retry@4.0.7/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
          at async <my project path>/node_modules/.pnpm/@aws-sdk+middleware-flexible-checksums@3.758.0/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:319:18
          at async <my project path>/node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.758.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:110:22
          at async <my project path>/node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.758.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:137:14
  "name": "MissingContentMD5",
  "$fault": "client",
  "$metadata": {
    "httpStatusCode": 400,
    "requestId": "182A4BCFF6AFA0C7",
    "extendedRequestId": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8",
    "attempts": 1,
    "totalRetryDelay": 0
  },
  "Code": "MissingContentMD5",
  "BucketName": "<redacted bucket name>",
  "Resource": "/<redacted bucket name>/",
  "RequestId": "182A4BCFF6AFA0C7",
  "HostId": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8"
}

I've read the announcement thread on the recent S3 default integrity change, but that appears to only relate to GET and PUT calls.

I've also read the REST API docs on DeleteObjects, and that says clearly that:

The Content-MD5 request header is required for all Multi-Object Delete requests. Amazon S3 uses the header value to ensure that your request body has not been altered in transit.

I've tried setting ChecksumAlgorithm to something else, like Sha256, but that doesn't fix this issue. And I don't see any other param or option to opt in to calculating an MD5 for this request.

I'm pretty sure that calculating the MD5 of the request, and sending this Content-Md5 header, should be done automatically by the SDK here, since it's required for all calls, right?

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-s3@3.758.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v22.9.0

Reproduction Steps

See code snippet above.

Observed Behavior

See error above.

Expected Behavior

No error.

Possible Solution

No response

Additional Information/Context

No response

[aseemk]

Hi there! Just checking if any update or confirmation. (We're blocked on this issue; will just be helpful to know whether I should invest in a workaround or downgrade back to an older version of this library, etc.) Thank you!

[aBurmeseDev]

Hi @aseemk - apologies for the delay here.

This's indeed related to recent change with S3 default integrity. We do have a supplemental doc for this: https://github.com/aws/aws-sdk-js-v3/blob/main/supplemental-docs/MD5_FALLBACK.md

Let me know if that helps!

[aseemk]

No worries re: delay and thanks for the response @aBurmeseDev.

I'm really confused though: I'm using S3 itself, not any third-party S3-compatible service. Is it really expected that DeleteObjects doesn't work against S3 itself, with this latest SDK??

[aseemk]

Oh, I'm sorry, I realize now that when I'm testing and repro'ing this locally, it's actually against minio, not real S3. My apologies!

That said, I continue to get this error against Minio even with the MD5 fallback client:

{
  "type": "S3ServiceException",
  "message": "Missing required header for this request: Content-Md5.",
  "stack":
      MissingContentMD5: Missing required header for this request: Content-Md5.
          at throwDefaultError (<my project path>/node_modules/.pnpm/@smithy+smithy-client@4.1.6/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)
          at <my project path>/node_modules/.pnpm/@smithy+smithy-client@4.1.6/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5
          at de_CommandError (<my project path>/node_modules/.pnpm/@aws-sdk+client-s3@3.758.0/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4971:14)
          at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
          at async <my project path>/node_modules/.pnpm/@smithy+middleware-serde@4.0.2/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
          at async <anonymous> (<my project path>/api/src/aws/S3ClientWithMD5.ts:29:24)
          at async <my project path>/node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.758.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:484:18
          at async <my project path>/node_modules/.pnpm/@smithy+middleware-retry@4.0.7/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
          at async <my project path>/node_modules/.pnpm/@aws-sdk+middleware-flexible-checksums@3.758.0/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:319:18
          at async <my project path>/node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.758.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:110:22
  "name": "MissingContentMD5",
  "$fault": "client",
  "$metadata": {
    "httpStatusCode": 400,
    "requestId": "182C45A53916B707",
    "extendedRequestId": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8",
    "attempts": 1,
    "totalRetryDelay": 0
  },
  "Code": "MissingContentMD5",
  "BucketName": "<my bucket name>",
  "Resource": "/<my bucket name>/",
  "RequestId": "182C45A53916B707",
  "HostId": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8"
}

The stack trace shows it's using the MD5 client:

at async <anonymous> (<my project path>/api/src/aws/S3ClientWithMD5.ts:29:24)

The only thing I've changed from the reference code is that I do still need to pass custom configuration to the underlying S3 client:

// Before:
const client = new S3Client({});

// After:
const client = new S3Client({
  endpoint: 'http://localhost:9000/', // Minio endpoint
  forcePathStyle: true, // since Minio
  region: 'us-west-2',
  credentials: undefined, // default to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars
});

What would cause even the reference code to keep returning this error? Thanks!

[aseemk]

Okay:

1. Turns out Minio has updated to support these new checksums. Support AWS S3 new checksums minio/minio#20845. (I just needed to update/re-pull my local Docker image.)

2. The reference code for createS3ClientWithMD5 has two bugs:

   1. Can't keep reusing a single MD5 hash instance across requests. Need to call createHash('md5') each request.
   2. Wrong to call await next(args) before updating args.request.headers. Need to call it after.

FYI on the latter — might be worth fixing for others.

Thank you for the pointer and help! I'm all set now.

[siddsriv]

Thanks for reporting, glad you're not blocked anymore! We'll take a look and put out the hash instance fix shortly.

[Zmaon]

For those interested, here is a modified (TypeScript) version of the provided createS3ClientWithMD5 that works for me:

/**
 * Fixed version from https://github.com/aws/aws-sdk-js-v3/blob/main/supplemental-docs/MD5_FALLBACK.md
 * This is because the AWS SDK for JavaScript v3 uses CRC32 checksums by default, but this doesn't work
 * with other providers like Infomaniak Public Cloud, that still requires MD5 checksums.
 */

import { S3Client } from '@aws-sdk/client-s3';
import { HttpRequest } from '@smithy/types';
import { createHash } from 'crypto';

type Args = {
  request: HttpRequest;
};

/**
 * Creates an S3 client that uses MD5 checksums for DeleteObjects operations
 */
export function createS3ClientWithMD5(configuration = {}) {
  const client = new S3Client(configuration);

  // Add middleware to remove any SDK-added checksums and add MD5 instead
  // This happens at the 'build' stage, before the request is signed
  client.middlewareStack.add(
    (next, context) => async (args) => {
      const typedArgs = args as Args;

      // Check if this is a DeleteObjects command
      const isDeleteObjects = context.commandName === 'DeleteObjectsCommand';
      if (!isDeleteObjects) {
        return next(args);
      }

      const headers = typedArgs.request.headers;

      // Remove any checksum headers
      Object.keys(headers).forEach((header) => {
        if (
          header.toLowerCase().startsWith('x-amz-checksum-') ||
          header.toLowerCase().startsWith('x-amz-sdk-checksum-')
        ) {
          delete headers[header];
        }
      });

      // Add MD5
      if (typedArgs.request.body) {
        const bodyContent = Buffer.from(typedArgs.request.body);
        // Create a new hash instance for each request
        headers['Content-MD5'] = createHash('md5').update(bodyContent).digest('base64');
      }

      return await next(args);
    },
    {
      step: 'build',
    }
  );
  return client;
}