`Upload` utility class API should allow us to have `FULL_OBJECT` checksum check

[kasir-barati]

Describe the feature

I need to check the uploaded file's integrity. I get an stream of data, so I was also hoping to see a more intuitive API. I mean right now I have to calculate the checksum of the entire file first since I need it when I am instantiating a new instance of this class.

It is somewhat related to this issue: #6742

Use Case

What this means in a real world application is that my client has to read the file twice which does not make sense;

1. I am gonna ask for the checksum (need it here):
2. They will stream the file!

new Upload({
  ...,
  params: {
    ChecksumCRC32: "some checksum",
    Body: passThroughStream
  }
})

• As I've already mentioned it, this code as it is will lead to errors due to how you're just passing all the params in each request:

  • aws-sdk-js-v3/lib/lib-storage/src/Upload.ts

    Line 124 in 1f1905e

    const params = { ...this.params, Body: dataPart.data };

  • aws-sdk-js-v3/lib/lib-storage/src/Upload.ts

    Line 205 in 1f1905e

    const createCommandParams = { ...this.params, Body: undefined };

  • Especially this one, it is basically telling AWS S3 to generate a checksum for each part whereas what I want is a checksum for the whole object.

So it would be great if we could just pass the checksum for a FULL_OBJECT checksum check at the end.

Proposed Solution

new Upload({
  ...,
  params: {
    ChecksumCRC32: "some checksum",
    Body: passThroughStream,
    ChecksumType: "FULL_OBJECT",
  }
})

So now here and other places you can do a simple check, if ChecksumType is FULL_OBJECT you can skip attaching this params to the request.

Same can be applied to the rest of the places you're simply spreading whatever developer passes to the params.

Other Information

I like to contribute but to me your codebase looked like it's been generated from a python code. So I am not entirely sure how this SDK is being developed and maintained.

Note

Please take a look at my repo: https://github.com/kasir-barati/bugs/tree/aws-s3-multipart-completion-error

There you can see how ugly Upload's interface is right now. Especially this example: https://github.com/kasir-barati/bugs/blob/1149c73557b939a911ec4ec999c8430aa99124f0/upload1.ts

And I really like to know when do you think you can address this feature request.

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

SDK version used

3.749.0

Environment details (OS name and version, etc.)

Linux, Ubuntu 24.04.01

[kasir-barati]

Workarounds

1. Using middleware: https://github.com/kasir-barati/bugs/blob/db72dbcb8f2b9c832370ea08c063b80d69912d56/upload1.ts

2. This is the modified version which is doing the job but it is dirty and just a POC: kasir-barati/bugs@cdd3342

   And this is how you can use it: https://github.com/kasir-barati/bugs/blob/db72dbcb8f2b9c832370ea08c063b80d69912d56/upload2.ts#L72-L74

[zshzbh]

Thanks for the feedback. I will forward this to the service team.

@kasir-barati Thanks for providing the workaround!

[kasir-barati]

And just a few more comments, right now my client has to send the data chunks in small chunks (like 1MB at max) and I know that AWS S3 requires my code to upload at least 5MB chunk size each time I call UploadPartCommand. And this is all good and well but it will prevent use from being able to retry to upload a chunk again.

... If transmission of any part fails, you can retransmit that part without affecting other parts ...

— ref.

So basically we loss this functionality and this can be a real issue in case of a huge file with a bad network condition. And to top it off assume it is a critical situation where we cannot retry to upload the whole file, we can only retry the failing parts. What we're doing right now is sending a chunk number for each request. But I wonder how I can associate those chunk numbers for each 1MB to the ones Upload is sending to the AWS.

So this would be a great feature to have. Please understand that my client will upload the file in a stream to my backend, thus it needs to generate a checksum for the file. And since the chunk size is less than 1Mb I had to accumulate those chunks and send them as they reaches sizes bigger than 5MB.

And now my client is computing the checksum of the entire file with a CRC based algorithm to do a FULL_OBJECT checksum type check at the very end. With that said what I like to see is a way to enable my client to retry uploading a single chunk agai without causing corruption in my data.

Any comment?