Nextflow exits with AwsBatch executor - S3 multipart upload error

[tamuanand]

Hi

I am using NF 21.10.4 with awsbatch executor. Halfway thru my workflow, the program exits with a S3 multipart upload error - See details below

N E X T F L O W version 21.10.4 build 5656
  created 30-11-2021 16:09 UTC

My nextflow script

Channel
    .fromPath(params.input_x_predictor)
    .set { input_csv_x_predictor_ch }

Channel
    .fromPath(params.input_y_target)
    .set { input_csv_y_target_ch }

process splitCSV_multiple_files {

 // split a big CSV file (1000 rows, 16k columns) 
 // create 16k files each with 2 columns 
 
 cpus 24
  memory '200 GB'

  input: file y_target from input_csv_y_target_ch

  output: path('*.csv') into predict_ml_ch

  script:
  """
    awk -F, 'BEGIN{OFS=",";} {for (i=1; i < NF; i++) print \$1, \$(i+1) > i ".csv"}' $y_target
  """
}

process predict_ML {

 // for each 2 column file from above process, run this 

  cpus 24
  memory '200 GB'
  
input:
  file x_predictor from input_csv_x_predictor_ch.collect()
  file split_csv_file from predict_ml_ch.flatten()

  output:
  path('*_prediction.csv') into merged_output_ch

  script:
  """
  pyscript.py $x_predictor $split_csv_file \
      "${split_csv_file.SimpleName}_prediction.csv"
  """
}

Error messages causing exit

Upload: ugsbvWYD45UlaDEWzXpPuyNUWEBUOf9icwS58UDD7HhQe1HdMLByexLtpuAkIck9hhrFC8_KGHrRpRHILhMCYVJuJrQrHjGxiGKZyJy2E5sEHTLQsT5vWDWZOVeAkJAQqda9NqEyuTxI2wkvI3lgnbXP114hasg2bhTmbNnTPnU- > Error for part: 10003
Caused by: java.io.IOException: Failed to upload multipart data to Amazon S3
        at com.upplication.s3fs.S3OutputStream.uploadPart(S3OutputStream.java:447)
        at com.upplication.s3fs.S3OutputStream.access$000(S3OutputStream.java:66)
        at com.upplication.s3fs.S3OutputStream$1.run(S3OutputStream.java:348)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Part number must be an integer between 1 and 10000, inclusive (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 7W2GEESJCXZJK0HM; S3 Extended Request ID: eKzYtuqtr115DjtSQXg1gAUeAvKqlZZ6U68UfXM2lNYx9JCXcWCLs8l/omN4u7XFQfizFq7Lur4=; Proxy: null), S3 Extended Request ID: eKzYtuqtr115DjtSQXg1gAUeAvKqlZZ6U68UfXM2lNYx9JCXcWCLs8l/omN4u7XFQfizFq7Lur4=
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384)
        at com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3875)
        at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3860)
        at com.upplication.s3fs.S3OutputStream.uploadPart(S3OutputStream.java:480)
        at com.upplication.s3fs.S3OutputStream.uploadPart(S3OutputStream.java:442)
        ... 7 more

Upload: ugsbvWYD45UlaDEWzXpPuyNUWEBUOf9icwS58UDD7HhQe1HdMLByexLtpuAkIck9hhrFC8_KGHrRpRHILhMCYVJuJrQrHjGxiGKZyJy2E5sEHTLQsT5vWDWZOVeAkJAQqda9NqEyuTxI2wkvI3lgnbXP114hasg2bhTmbNnTPnU- > Error for part: 10022
Caused by: java.io.IOException: Failed to upload multipart data to Amazon S3
        at com.upplication.s3fs.S3OutputStream.uploadPart(S3OutputStream.java:447)
        at com.upplication.s3fs.S3OutputStream.access$000(S3OutputStream.java:66)
        at com.upplication.s3fs.S3OutputStream$1.run(S3OutputStream.java:348)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Part number must be an integer between 1 and 10000, inclusive (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: 7W2QW8NT00DY7SQV; S3 Extended Request ID: v6H5k4a6xh1ui9G2bvtGV1Pz1ccpXh5bPV+txTM2vi84fH7gxIHPgl53RQarNTlGvVrxFy8hX5c=; Proxy: null), S3 Extended Request ID: v6H5k4a6xh1ui9G2bvtGV1Pz1ccpXh5bPV+txTM2vi84fH7gxIHPgl53RQarNTlGvVrxFy8hX5c=
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384)
        at com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3875)
        at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3860)
        at com.upplication.s3fs.S3OutputStream.uploadPart(S3OutputStream.java:480)
        at com.upplication.s3fs.S3OutputStream.uploadPart(S3OutputStream.java:442)
        ... 7 more

How do I get around this issue? Do I have to do something to control the number of simultaneous jobs (is that the issue?)

Thanks in advance.

[AaronNHart]

Sorry to resurrect a dead issue but it's high in my google search on the topic and it seems a few others have run into this. Was there any outcome?

[AaronNHart]

I would add that looking at the log file for me at least, it was failing due to trying to upload to many parts to a multipart upload.

[bentsherman]

I believe this issue has been fixed since 22.10, see #3113