Job definition error for AWS Batch job

[avw-adifranco]

Hi,

I've been trying to adapt my pipeline to work on AWS Batch but the pipeline keeps on failing and I have some troubles to figure out where the issue is.

My goal at the moment is to run the nextflow instance from my computer and send the jobs to AWS batch.
I've created an ECR image containing my software working from local docker as well as when pulling it from ECR (after granting permission to docker through aws cli). I've also followed the aws-genomics-workflows documentation to build a proper batch environnement with queues, EC instances, S3 bucket, roles and so on.

When I run my pipeline I get the following error :

Caused by:
  Invalid AWS Batch job definition -- provide a Docker image name or a Batch job definition name

with the following command and config

nextflow -C aws.config run main.nf --in Cluster_000006.fna -bucket-dir s3://mybucket/work

plugins {
        id 'nf-amazon'
}

process {
        executor = 'awsbatch'
        withlabel: gapseq {
                container = 'arn of my jobDefinition or of my ECR container'
                cpus = 1
        }
        withname: find {
                cpus = 1
        }
        queue = 'arn of my defined queue'
        //containerOptions = '--shm-size 16000000 --ulimit nofile=1280:2560 --ulimit nproc=16:32'
}

aws {
        region = 'eu-west-3'
        batch {
                cliPath = '/home/ec2-user/miniconda/bin/aws'
        }
}

I do not know from which side I should take the problem. Does the issue came from my container ? from how I configured the job definition using this container ? Is it an authentification problem (I have adminRights and it seems like my key is properly read from what I am seeing in my log).

The weird thing is that the job properly stage my infile to the bucket as well as writing the work dir but nothing much is coming from it.

If someone could share some light on this one, it would be helpful.

Also here is the nextflow.log

févr.-28 14:59:51.544 [main] DEBUG nextflow.cli.Launcher - $> nextflow -C aws.config run main.nf --in Cluster_000006.fna -bucket-dir 's3://gwfcore-aviwell-bucket/work/'
févr.-28 14:59:51.599 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 22.10.4
févr.-28 14:59:51.615 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/home/adf/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
févr.-28 14:59:51.621 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
févr.-28 14:59:51.622 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
févr.-28 14:59:51.624 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
févr.-28 14:59:51.634 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
févr.-28 14:59:51.648 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/adf/Aviwell/dev/nextflow_gapseq_AWS/aws.config
févr.-28 14:59:51.667 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
févr.-28 14:59:52.069 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
févr.-28 14:59:52.079 [main] INFO  nextflow.cli.CmdRun - Launching `main.nf` [tender_brattain] DSL2 - revision: 720701a47f
févr.-28 14:59:52.081 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins declared=[[email protected]]
févr.-28 14:59:52.081 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[[email protected]]
févr.-28 14:59:52.081 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[[email protected]]
févr.-28 14:59:52.081 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-amazon version: 1.11.2
févr.-28 14:59:52.088 [main] INFO  org.pf4j.AbstractPluginManager - Plugin '[email protected]' resolved
févr.-28 14:59:52.088 [main] INFO  org.pf4j.AbstractPluginManager - Start plugin '[email protected]'
févr.-28 14:59:52.100 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started [email protected]
févr.-28 14:59:52.112 [main] DEBUG nextflow.file.FileHelper - > Added 'S3FileSystemProvider' to list of installed providers [s3]
févr.-28 14:59:52.122 [main] DEBUG nextflow.secret.LocalSecretsProvider - Secrets store: /home/adf/.nextflow/secrets/store.json
févr.-28 14:59:52.125 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@1b1c538d] - activable => nextflow.secret.LocalSecretsProvider@1b1c538d
févr.-28 14:59:52.167 [main] DEBUG nextflow.Session - Session UUID: 9aac9d32-ea3e-45c3-ab38-c519eb3a7fc8
févr.-28 14:59:52.167 [main] DEBUG nextflow.Session - Run name: tender_brattain
févr.-28 14:59:52.167 [main] DEBUG nextflow.Session - Executor pool size: 16
févr.-28 14:59:52.174 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=48; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
févr.-28 14:59:52.189 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 22.10.4 build 5836
  Created: 09-12-2022 09:58 UTC (10:58 CEST)
  System: Linux 5.19.0-32-generic
  Runtime: Groovy 3.0.13 on OpenJDK 64-Bit Server VM 11.0.18+10-post-Ubuntu-0ubuntu122.04
  Encoding: UTF-8 (UTF-8)
  Process: 48076@TLS-TWB-PO214 [127.0.1.1]
  CPUs: 16 - Mem: 31.1 GB (21.5 GB) - Swap: 2 GB (2 GB)
févr.-28 14:59:52.201 [main] DEBUG nextflow.Session - Work-dir: /home/adf/Aviwell/dev/nextflow_gapseq_AWS/work [ext2/ext3]
févr.-28 14:59:52.207 [main] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider
févr.-28 14:59:52.215 [main] DEBUG nextflow.Global - Using AWS credential defined in `default` section in file: /home/adf/.aws/credentials
févr.-28 14:59:52.216 [main] DEBUG nextflow.file.FileHelper - AWS S3 config details: {secret_key=8soraY.., region=eu-west-3, max_error_retry=5, access_key=AKIAQ2..}
févr.-28 14:59:52.498 [main] DEBUG nextflow.Session - Bucket-dir: s3://gwfcore-aviwell-bucket/work
févr.-28 14:59:52.499 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /home/adf/Aviwell/dev/nextflow_gapseq_AWS/bin
févr.-28 14:59:52.513 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[AwsBatchExecutor]
févr.-28 14:59:52.522 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
févr.-28 14:59:52.535 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
févr.-28 14:59:52.542 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 17; maxThreads: 1000
févr.-28 14:59:52.610 [main] DEBUG nextflow.Session - Session start
févr.-28 14:59:52.892 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
févr.-28 14:59:52.997 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: awsbatch
févr.-28 14:59:52.997 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'awsbatch'
févr.-28 14:59:53.001 [main] DEBUG nextflow.executor.Executor - [warm up] executor > awsbatch
févr.-28 14:59:53.013 [main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-executor, limiter:RateLimiter[stableRate=50.0qps], poolSize:80, maxPoolSize:80, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
févr.-28 14:59:53.018 [main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-reaper, limiter:RateLimiter[stableRate=50.0qps], poolSize:80, maxPoolSize:80, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
févr.-28 14:59:53.018 [main] DEBUG n.cloud.aws.batch.AwsBatchExecutor - Creating parallel monitor for executor 'awsbatch' > pollInterval=10s; dumpInterval=5m
févr.-28 14:59:53.044 [main] DEBUG nextflow.Global - Using AWS credential defined in `default` section in file: /home/adf/.aws/credentials
févr.-28 14:59:53.117 [main] DEBUG n.cloud.aws.batch.AwsBatchExecutor - [AWS BATCH] Executor options=AwsOptions(cliPath:/home/ec2-user/miniconda/bin/aws, storageClass:null, storageEncryption:null, storageKmsKeyId:null, remoteBinDir:null, region:eu-west-3, maxParallelTransfers:4, maxTransferAttempts:5, delayBetweenAttempts:10s, retryMode:standard, maxSpotAttempts:5, debug:null, fetchInstanceType:false, jobRole:null, logsGroup:null, volumes:[], shareIdentifier:null, awsCli:/home/ec2-user/miniconda/bin/aws --region eu-west-3)
févr.-28 14:59:53.171 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: awsbatch
févr.-28 14:59:53.171 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'awsbatch'
févr.-28 14:59:53.175 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: awsbatch
févr.-28 14:59:53.175 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'awsbatch'
févr.-28 14:59:53.179 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: awsbatch
févr.-28 14:59:53.179 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'awsbatch'
févr.-28 14:59:53.182 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: awsbatch
févr.-28 14:59:53.182 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'awsbatch'
févr.-28 14:59:53.184 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: find, gapseq, transporter, draft, medium, fill
févr.-28 14:59:53.184 [main] DEBUG nextflow.Session - Igniting dataflow network (6)
févr.-28 14:59:53.188 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > find
févr.-28 14:59:53.189 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > transporter
févr.-28 14:59:53.189 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > draft
févr.-28 14:59:53.189 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > medium
févr.-28 14:59:53.190 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > fill
févr.-28 14:59:53.191 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination 
févr.-28 14:59:53.192 [main] DEBUG nextflow.Session - Session await
févr.-28 14:59:54.020 [FileTransfer-1] DEBUG nextflow.file.FilePorter - Local cache found for foreign file /home/adf/Aviwell/dev/nextflow_gapseq_AWS/Cluster_000006.fna at s3://gwfcore-aviwell-bucket/work/stage/fd/3e0412e5cf68b12889b6a8417918b1/Cluster_000006.fna
févr.-28 14:59:54.417 [AWSBatch-executor-1] DEBUG nextflow.processor.TaskRun - Unable to dump error of process 'null' -- Cause: java.nio.file.NoSuchFileException: /tmp/temp-s3-6250664469596032815/.command.log
févr.-28 14:59:54.418 [AWSBatch-executor-1] ERROR nextflow.processor.TaskProcessor - Error executing process > 'transporter (1)'

Caused by:
  Invalid AWS Batch job definition -- provide a Docker image name or a Batch job definition name

Command executed:

  gapseq find-transport -b 200 Cluster_000006.fna

Command exit status:
  -

Command output:
  (empty)

Work dir:
  s3://gwfcore-aviwell-bucket/work/44/b5b8f79f9cd86968e1a76efc9b3824

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
févr.-28 14:59:54.421 [AWSBatch-executor-1] DEBUG nextflow.Session - Session aborted -- Cause: Invalid AWS Batch job definition -- provide a Docker image name or a Batch job definition name
févr.-28 14:59:54.431 [AWSBatch-executor-1] DEBUG nextflow.Session - The following nodes are still active:
[process] draft
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: inputFna
  port 1: (queue) OPEN  ; channel: __$pathinparam<1>
  port 2: (queue) OPEN  ; channel: __$pathinparam<2>
  port 3: (queue) OPEN  ; channel: __$pathinparam<3>
  port 4: (cntrl) -     ; channel: $

[process] medium
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: inputFna
  port 1: (queue) OPEN  ; channel: __$pathinparam<1>
  port 2: (queue) OPEN  ; channel: __$pathinparam<2>
  port 3: (cntrl) -     ; channel: $

[process] fill
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: inputFna
  port 1: (queue) OPEN  ; channel: __$pathinparam<1>
  port 2: (queue) OPEN  ; channel: __$pathinparam<2>
  port 3: (queue) OPEN  ; channel: __$pathinparam<3>
  port 4: (queue) OPEN  ; channel: __$pathinparam<4>
  port 5: (cntrl) -     ; channel: $

févr.-28 14:59:54.432 [main] DEBUG nextflow.Session - Session await > all processes finished
févr.-28 14:59:54.432 [main] DEBUG nextflow.Session - Session await > all barriers passed
févr.-28 14:59:54.436 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=2; ignoredCount=0; cachedCount=0; pendingCount=2; submittedCount=0; runningCount=-2; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=0ms; cachedDuration=0ms;loadCpus=-2; loadMemory=0; peakRunning=0; peakCpus=0; peakMemory=0; ]
févr.-28 14:59:54.637 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
févr.-28 14:59:54.637 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin '[email protected]'
févr.-28 14:59:54.638 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-amazon
févr.-28 14:59:54.654 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

[bentsherman]

I think the problem is that you have to define a "default" container via process.container, even if all of your processes are covered via withName / withLabel. Also, not sure if withName / withLabel are case-sensitive ...

[avw-adifranco]

Hi Ben,

Thanks for your highlight ! It was effectively the reason !

I would never have guess it from this error message. I would have expected to get a 'Warning: no default container for a AWS batch executor' kind of message.

By moving my container outside my withLabel entry, I got the aws not found error, fixed my ECR and finally manage to get it work.

Thanks again.