AWS s3 upload not complete

[hurrialice]

Hi nextflow team, thanks for the great work!

I have been running nextflow on HPC and upload results to a s3 bucket:

publishDir file(params.outdir), mode: "copy", saveAs: { filename -> "${sample_name}_$filename" }

while params.outdir is a s3 bucket with write access provided by default aws profile.

My aws scope in nextflow.config looks like this:

aws {
    client {
        maxConnections = 50
        maxErrorRetry = 5
        connectionTimeout = 10000
        socketTimeout = 10000
        storageEncryption = 'AES256'
        downloadParallel = true
        downloadChunkSize = '50 MB'
        downloadNumWorkers = 10
        downloadMaxAttempts = 5
        uploadMaxThreads = 10
        uploadRetrySleep = '1500ms'
        uploadChunkSize = '50 MB'
    }
}

However when I have multiple large files (e.g. BAMs) to upload concurrently, I often get incomplete files in s3. In a recent test with 11 concurrent BAM (each is about 50G) uploads, I got 5 partial BAM files -- notably, each partial BAM file is either 500MB or 550MB. I wonder if you have any suggestion on tweaking my config above. I was suspecting the upload was bottlenecked by uploadMaxThreads*uploadChunkSize but I wasn't sure... I did not find anything useful from .nextflow.log

Thanks for your time! Any suggestion is greatly appreciated!

[hurrialice]

I think my major confusion is that I don't see any error message on log files and the problem was only picked up when I tried to load those BAMs to IGV. I wonder if there is anything I can do to make sure the upload is robust -- e.g. compare hashes of my local files vs s3 uploaded files? Thanks for any insights in advance.