Memory leak issue with `Upload` class when uploading large files (more than 200 MB)

[kasir-barati]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

I am using this SDK and when my app tries to upload a really large file I can see that the memory consumption is increasing and then we have also spikes.

Note

This is what I can see in portainer, but for local env I am just relying on logs and docker stats command.

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-s3@3.749.0, @aws-sdk/lib-storage@3.749.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

NodeJS: 22.12.0

Reproduction Steps

Clone this repo and follow these instructions.

Observed Behavior

Docker kills the container since it exceeds the memory limitations.

Expected Behavior

I am not sure if this is the intended behavior of this class and if I am using Upload class correctly.

But I expects it to just work. Or maybe I should not use PathThrough 😕.

Possible Solution

IDK.

Additional Information/Context

What I hope to see is that I can still use Upload class without this memory issue.

[kasir-barati]

Note

IDK why and I would love to learn why it works when I pass the returned value from createReadStream directly to params.Body. So basically it can upload a 250 MB file to S3:

https://github.com/kasir-barati/bugs/blob/8a34fa3b1c94dc739dd95f521ac79a7e7ba87b8c/main.ts#L101-L119

Any idea?

Caution

In my real app I have a duplex gRPC stream which I use to upload a file. That's why I am using PathThough. I get chunks of data and I need to then upload them to AWS S3 chunk by chunk. But apparently PathThrough holds onto the previously uploaded chunks and so do the AWS S3 😭.

[kasir-barati]

What am I doing wrong?

How I can use PathThrough or something similar while not having this memory leak issue?

Note

I tried now to remove the memory limits in compose file and the peak of memory usage was ~650 MB! But this code did not work 🤯!

[aBurmeseDev]

Hi @kasir-barati - thanks for checking in.

I'd like to gather a few details before looking into your code:

• If i understand correctly, this only happens for larger files (>200MB). Is that consistent?
• Does this only happen within your app? Any chance you can run this wit SDK only, outside your app?
• I noticed the SDK version you're using was released back in Feb, can you upgrade version and test?
• I don't think using PassThrough is the culprit but if you'd like alternative, you can try passing stream directly as Body as shown below:

import { S3Client } from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';

const upload = new Upload({
    client: new S3Client({}),
    params: {
        Bucket: bucket,
        Key: 'test.txt',
        Body: stream,
        ContentType: 'text/plain',
    },
});

upload.done();

[kasir-barati]

If i understand correctly, this only happens for larger files (>200MB). Is that consistent?

Yes

Does this only happen within your app? Any chance you can run this wit SDK only, outside your app?

Not sure, I tried to simulate it in my repo.

I noticed the SDK version you're using was released back in Feb, can you upgrade version and test?

No. Though I think version is irrelevant.

I don't think using PassThrough is the culprit but if you'd like alternative, you can try passing stream directly as Body as shown below:

I cannot see how I can do it since I will receive the data as a gRPC duplex stream:

syntax = "proto3";

package File;

service FileUploadService {
  rpc Upload(stream Chunk) returns (stream UploadResponse) {}
}

message Chunk {
  optional string file_name = 1;
  optional float total_size = 2;
  // This will be sent with the last chunk of the file.
  optional string checksum = 3;
  int32 part_number = 4;
  // CRC32 and CRC32C are the only supported algorithms for FULL_OBJECT checksum check.
  optional string checksum_algorithm = 5;
  optional string id = 6;
  bytes data = 7;
}

message UploadResponse {
}