Support checksum algorithm for S3 uploads

[bentsherman]

Currently stalled because the Java SDK doesn't appear to support this option in the way that we need it:

https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3control/model/S3CopyObjectOperation.html#setChecksumAlgorithm-java.lang.String-

[pditommaso]

Can you provide more details on the goal? Nextflow computes the MD5 checksum when uploading S3 files

nextflow/plugins/nf-amazon/src/main/com/upplication/s3fs/S3OutputStream.java

Line 522 in a1d0846

request.setMd5Digest(Base64.encodeAsString(checksum));

[bentsherman]

Indeed, but our client wants to compute the SHA-256 checksum to meet some FDA requirements, so they need this new option.

[bentsherman]

To be clear, they could compute the checksum manually, and they do, but this option would automate it for them.

[pditommaso]

But if I'm understanding correctly the AWS SDK does not allow the specify it, does it?

[bentsherman]

It seems the Python SDK does but the Java SDK does not. The original blog post had this python example:

with open(file_path, 'rb') as file:
    r = s3.put_object(
        Bucket=bucket,
        Key=key,
        Body=file,
        ChecksumAlgorithm='sha1'
    )

I also just found it in the Java SDK v2: https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/model/PutObjectRequest.Builder.html#checksumAlgorithm(java.lang.String)

But it is not in v1.

[pditommaso]

Meh

[bentsherman]

I spoke to the AWS devs and they said that the checksum algorithm option is available in the AWS Java SDK v2 (see here). We'll have to migrate to v2 in order to use this feature.

[pditommaso]

that's a mess

[drernie]

We'll have to migrate to v2 in order to use this feature
We also will need this. Will nf-amazon v2 use the JDKv2?

[pditommaso]

We also will need this. Will nf-amazon v2 use the JDKv2?

nf-amazon is already out and it's still based on JDK v1

[bentsherman]

@jorgee when you have some time, can you try to implement this PR with SDK v2? This was a highly-requested feature that was blocked by the SDK v2 upgrade.

Happy to give more context if you need it. Feel free to create a new PR if you prefer.

[jorgee]

I have created #6321 with a first implementation of the checksum algorithm with AWS SDK v2. We can close it and continue the discussion in the new PR.