The 'total' property is always undefined when uploading files larger than 5MB

[ti2sugiyama]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

When upload large file using stream by Upload module on webApp, httpUploadProgress event object property 'total' always undefined.

SDK version number

@aws-sdk/lib-storage@3.264.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v18.13.0

Reproduction Steps

1. setup Node.js project
2. install @aws-sdk/client-s3 and @aws-sdk/lib-storage
3. prepare upload file (larger than 5 MB)
4. run the following program
5. may see ev.total always undefined

const fs = require("fs");
const stream = require("stream");
const S3Client = require("@aws-sdk/client-s3").S3Client;
const Upload = require("@aws-sdk/lib-storage").Upload;

const s3 = new S3Client({
  // your s3 settings
});

async function main() {
  // Use a file larger than 5MB.
  const fileStream = fs.createReadStream("/path/to/your/over5mb/file.dat");
  const passThrough = new stream.PassThrough();
  // Pipe PassThrough to remove fileStream.path
  // to emulate browser file upload.
  fileStream.pipe(passThrough);

  const upload = new Upload({
    client: s3,
    params: {
      Bucket: // your Bucket,
      Key: // your Key,
      Body: passThrough,
    },
  });

  upload.on("httpUploadProgress", function (ev) {
    // ev.total always undefined
    console.log(ev);
  });

  await upload.done();
}

main();

By debugging the code, no assignment for this.totalBytes( references of 'total') anywhere excepted in constructor and function for one chunk

Observed Behavior

httpUploadProgress event object property 'total' always undefined.
(In sample code, console output of ev.total is undefined to the end )

Expected Behavior

finally set the correct total file size
(In sample code, final console output of ev.total is the correct total file size)

Possible Solution

No response

Additional Information/Context

aws-sdk v2 was recalculating this.totalBytes on readable event
https://github.com/aws/aws-sdk-js/blob/b168eaab7aa04f8fc300b38bf67f9c26bf02c28a/lib/s3/managed_upload.js#L188

[yenfryherrerafeliz]

Hi @ti2sugiyama, thanks for opening this issue. I can confirm the reported behavior. I will investigate this further in order to define further steps on this.

Thanks!

[yenfryherrerafeliz]

@ti2sugiyama what you say in the comment below is true:

By debugging the code, no assignment for this.totalBytes( references of 'total') anywhere excepted in constructor and function for one chunk

I feel this issue just happens when the body given as parameter does not have any of the properties evaluated in the conditions here. As of right now I will mark this issue to be reviewed so it can be further addressed.

Steps to reproduce:

• Create an empty node project: npm init -y
• Add the dependencies:
  • npm i @aws-sdk/client-s3
  • npm i @aws-sdk/lib-storage
• Then, create an empty file. Ex: src/index.js
• Copy and paste into the file created above, the following code:

import {S3Client} from "@aws-sdk/client-s3";
import {Upload} from "@aws-sdk/lib-storage";
import * as fs from "fs";
import * as stream from "stream";

const client = new S3Client({
    region: process.env.TEST_REGION,
})
const pathToFile = "./test-file.txt"
const data = "#".repeat(1024 * 1024 * 10);
fs.writeFileSync(pathToFile, data);
const fileStream = fs.createReadStream(pathToFile)
const passThrough = new stream.PassThrough();
fileStream.pipe(passThrough)
const upload = new Upload({
    client: client,
    params: {
        Bucket: process.env.TEST_BUCKET,
        Key: process.env.TEST_KEY,
        Body: passThrough,
    }
})
upload.on('httpUploadProgress', (progress) => {
    console.log('Progress: ', progress)
})
await upload.done();

Thanks!

[devnomic]

Same issue. It's been a year, hopefully fixed soon.

[jirimoravcik]

Facing the same issue here.
@yenfryherrerafeliz suggested this approach in #5165, maybe there should be a note that it doesn't work for files larger than 5MB?

Also, is using max of all ev.loaded instead of ev.total a reasonable workaround for now? Thanks!