Damaged downloaded file

[waikiki-bn]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

Conditions:

1. Aws-sdk v3.972.0
2. Archiver v7.0.1
3. NodeJs v22.14.0

Currently I am working on migration from aws-sdk v2 to aws-sdk v3, but I have problems with downloading files. Files are corrupted after i get them from archive.

User case:

1. User choose file to download.
2. My node server with GetObjectCommand requests for data from s3.
3. Node server stream data from s3 to archive, which is streaming to response.

The main goal is to avoid saving files in memory and use streams.

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-s3: 3.972.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

22.14.0

Reproduction Steps

So here is example of my code

Download archive logic (error handlers are not included)

const archive = archiver.create('zip-encrypted', archiveOptions as ArchiverOptions)

response.setHeader('Content-Type', 'application/zip')
response.setHeader('Content-Disposition', "attachment; filename*=UTF-8''${suggestedArchiveName}")

const fileObjectReadStream = await s3Client.downloadAttachment({
    Bucket: s3.bucket,
    Key: s3.key
 })

 archive.append(fileObjectReadStream, { name })
 archive.finalize()

S3 downloadAttachment

    const downloadAttachment = async (
      params: GetObjectCommandInput,
      accessToken: string
    ): Promise<Readable> => {
      const command = new GetObjectCommand(params)
      const response = await this.s3Client.send(command)

      if (!response.Body) throw new Error('Missing file data')

      return response.Body as Readable
    }

Observed Behavior

I get the archive, but I cant open the file from the archive. The size of the unzipped file is larger than the size of the original file. (Original is 295.79 KB, unzipped file size is 299 KB)

this happens only with aws-sdk v3 version. I have tried to upload file with aws-sdk v3 and download it as an archive using aws-sdk v2 and everything is fine. The difference in code is to use return s3.upload(input, () => {}).promise() instead of return response Body

I've tried to use transformToWebStream but I've got "The stream has been consumed by other callback"
i've also tried to exlcude archive logic and download to fs as a test, file was damaged too.

Expected Behavior

Same result as in aws-sdk v2

Possible Solution

No response

Additional Information/Context

No response

[aBurmeseDev]

Hi @waikiki-bn - thanks for reaching out!

This's a documented breaking change in v3 with S3 GetObject responses where it returns Buffer in v2 but Stream in JSv3. See here: https://github.com/aws/aws-sdk-js-v3/blob/main/UPGRADING.md#amazon-s3

The thing is it's not returning a regular Node.js Readable stream but rather an SdkStream type (you can see this in the GetObjectCommand API docs). When you cast it with as Readable, TypeScript lets it slide but at runtime it's still not actually a Node.js stream. That's why archiver and file writes are getting corrupted that they're expecting a real Node.js stream with methods like .pipe().

Working with your current code, you might be able to just wrap it with Readable.from() to convert it to a proper Node.js stream:

import { Readable } from 'stream';

const downloadAttachment = async (
  params: GetObjectCommandInput,
  accessToken: string
): Promise<Readable> => {
  const command = new GetObjectCommand(params);
  const response = await this.s3Client.send(command);

  if (!response.Body) throw new Error('Missing file data');

  // Change this line:
  // return response.Body as Readable;
  return Readable.from(response.Body);
};

I've verified this change working on my end. Hope it helps and let me know if you run into any issues with this fix.

[waikiki-bn]

Hi @aBurmeseDev, I've tried this solution before and one more time after your replay, but a file is still corrupted

May be there are troubles with configuration?

Mine is

    this.s3Client = new S3Client({
      endpoint: 'mockEndpointHost',
      credentials: {
        accessKeyId: 'mockAccessKeyId',
        secretAccessKey: 'mockSecretAccessKey',
        sessionToken: ''
      },
      region: 'mockRegion',
      forcePathStyle: true
    })

[aBurmeseDev]

Hi! I don't think client config is the culprit for this stream type issue. Can you confirm you've changed the code to use Readable.from()? I want to make sure we're on the same page.

Your downloadAttachment should look like this:

  // Should be this:
  return Readable.from(response.Body);
  
  // NOT this:
  // return response.Body as Readable;
};

---

If yes and it's still corrupted, can you share your complete archive code how you're creating and finalizing the archive? Also what does "corrupted" mean exactly? Is it wrong file size, can't extract or content damaged?

[waikiki-bn]

H1, @aBurmeseDev!
Yes my downloadAttachment looks like

return Readable.from(response.Body);

I can't open any file and file size is large than the origin file size

I think there is no problems on JS side, may be there is a problem on s3 storage configuration or something else?
I hardly believe, that with about 16 million of downloading (npm statistics) only i have such problem