Process not getting executed on pod created by nextflow in K8s cluster

[raghuram-venugopal]

Bug report

I am running nextflow on k8s cluster which has 4 processess that require to run in 4 pods with a command similar to below.
nextflow run exec-pipeline.nf -c /storage-pvc-data/work/nextflow.config -profile dev

Expected Behavior: Process from nextflow should start and execute in the pod getting created via nextflow job.
Actual Behavior: Process in pod execution is failing to execute anything in the beginning with pod error as below.
Unknown option: -ue -- Check the available commands and options and syntax with 'help'

Steps to reproduce the problem

Due to restrictions from organization I would not be able to provide any code, but attaching the nextflow logs.

1. create any single process nf file
2. build a docker image to execute sample shell script
3. Create a k8s job to use command/args along with nextflow.config.yaml on pvc--> nextflow run exec-pipeline.nf -c /storage-pvc-data/work/nextflow.config -profile dev
4. Apply this job in K8s

Program output

N E X T F L O W  ~  version 23.10.1
Launching `exec-pipeline.nf` [elated_mirzakhani] DSL2 - revision: 770a423976
[-        ] process > process1             -
[-        ] process > process1_combination -

[-        ] process > process1             -
[-        ] process > process1_combination -
[-        ] process > process2      -
[-        ] process > process3          -

[-        ] process > process1             [  0%] 0 of 1
[-        ] process > process1_combination -
[-        ] process > process2      -
[-        ] process > process3          -

[-        ] process > process1             [  0%] 0 of 1
[-        ] process > process1_combination [  0%] 0 of 1
[-        ] process > process2      -
[-        ] process > process3          -

executor >  k8s (1)
[06/dcd3e7] process > process1             [  0%] 0 of 1
[-        ] process > process1_combination [  0%] 0 of 1
[-        ] process > process2      -
[-        ] process > process3          -

executor >  k8s (2)
[06/dcd3e7] process > process1             [  0%] 0 of 1
[1f/874414] process > process1_combination [  0%] 0 of 1
[-        ] process > process2      -
[-        ] process > process3          -

executor >  k8s (2)
[06/dcd3e7] process > process1             [  0%] 0 of 1
[1f/874414] process > process1_combination [  0%] 0 of 1
[-        ] process > process2      -
[-        ] process > process3          -

executor >  k8s (2)
[06/dcd3e7] process > process1             [  0%] 0 of 1
[1f/874414] process > process1_combination [  0%] 0 of 1
[-        ] process > process2      -
[-        ] process > process3          -

executor >  k8s (2)
[06/dcd3e7] process > process1             [  0%] 0 of 1
[1f/874414] process > process1_combination [  0%] 0 of 1
[-        ] process > process2      -
[-        ] process > process3          -
ERROR ~ Error executing process > 'process1'

Caused by:
  Process `process1` terminated for an unknown reason -- Likely it has been terminated by the external system

Command executed:

< masking the command due to restrictions >

Command exit status:
  -

Command output:
  (empty)

Command wrapper:
  Unknown option: -ue -- Check the available commands and options and syntax with 'help'

Work dir:
  /storage-pvc-data/work/06/dcd3e7dd3afdafe0ff7fd2b3a02d51

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details

executor >  k8s (2)
[06/dcd3e7] process > process1             [100%] 1 of 1, failed: 1 ✘
[1f/874414] process > process1_combination [  0%] 0 of 1
[-        ] process > process2      -
[-        ] process > process3          -
ERROR ~ Error executing process > 'process1'

Caused by:
  Process `process1` terminated for an unknown reason -- Likely it has been terminated by the external system

Command executed:

< masking the command due to restrictions >

Command exit status:
  -

Command output:
  (empty)

Command wrapper:
  Unknown option: -ue -- Check the available commands and options and syntax with 'help'

Work dir:
  /storage-pvc-data/work/06/dcd3e7dd3afdafe0ff7fd2b3a02d51

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details

executor >  k8s (2)
[06/dcd3e7] process > process1             [100%] 1 of 1, failed: 1 ✘
[1f/874414] process > process1_combination [  0%] 0 of 1
[-        ] process > process2      -
[-        ] process > process3          -
Error: check logs for detailed error
ERROR ~ Error executing process > 'process1'

Caused by:
  Process `process1` terminated for an unknown reason -- Likely it has been terminated by the external system

Command executed:

< masking the command due to restrictions >

Command exit status:
  -

Command output:
  (empty)

Command wrapper:
  Unknown option: -ue -- Check the available commands and options and syntax with 'help'

Work dir:
  /storage-pvc-data/work/06/dcd3e7dd3afdafe0ff7fd2b3a02d51

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
Pipelines completed

executor >  k8s (2)
[06/dcd3e7] process > process1             [100%] 1 of 1, failed: 1 ✘
[1f/874414] process > process1_combination [  0%] 0 of 1
[-        ] process > process2      -
[-        ] process > process3          -
Error: check logs for detailed error
Error: Pipeline execution stopped with the following message: null
ERROR ~ Error executing process > 'process1'

Caused by:
  Process `process1` terminated for an unknown reason -- Likely it has been terminated by the external system

Command executed:

< masking the command due to restrictions >

Command exit status:
  -

Command output:
  (empty)

Command wrapper:
  Unknown option: -ue -- Check the available commands and options and syntax with 'help'

Work dir:
  /storage-pvc-data/work/06/dcd3e7dd3afdafe0ff7fd2b3a02d51

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
WARN: Killing running tasks (1)

Feb-05 07:31:45.730 [main] DEBUG nextflow.cli.Launcher - $> nextflow run exec-pipeline.nf -c /storage-pvc-data/work/nextflow.config -profile dev
Feb-05 07:31:47.343 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 23.10.1
Feb-05 07:31:47.930 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/opt/nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Feb-05 07:31:48.232 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Feb-05 07:31:48.330 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Feb-05 07:31:48.437 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Feb-05 07:31:48.631 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Feb-05 07:31:48.937 [main] DEBUG nextflow.config.ConfigBuilder - User config file: /storage-pvc-data/work/nextflow.config
Feb-05 07:31:49.028 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /storage-pvc-data/work/nextflow.config
Feb-05 07:31:49.528 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `dev`
Feb-05 07:32:03.534 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [prod, dev, master]
Feb-05 07:32:04.032 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Feb-05 07:32:04.528 [main] INFO  nextflow.cli.CmdRun - Launching `exec-pipeline.nf` [elated_mirzakhani] DSL2 - revision: 770a423976
Feb-05 07:32:04.533 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Feb-05 07:32:04.534 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Feb-05 07:32:04.731 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /opt/nextflow/secrets/store.json
Feb-05 07:32:04.740 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@1df1ced0] - activable => nextflow.secret.LocalSecretsProvider@1df1ced0
Feb-05 07:32:06.029 [main] DEBUG nextflow.Session - Session UUID: 9f91d65a-18d3-457d-afee-ea59639d5e62
Feb-05 07:32:06.029 [main] DEBUG nextflow.Session - Run name: elated_mirzakhani
Feb-05 07:32:06.030 [main] DEBUG nextflow.Session - Executor pool size: 2
Feb-05 07:32:06.144 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Feb-05 07:32:06.239 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=10; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Feb-05 07:32:07.133 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 23.10.1 build 5891
  Created: 12-01-2024 22:01 UTC 
  System: Linux 5.10.205-195.807.amzn2.x86_64
  Runtime: Groovy 3.0.19 on OpenJDK 64-Bit Server VM 11.0.21+9-post-Ubuntu-0ubuntu122.04
  Encoding: UTF-8 (UTF-8)
  Process: 7@potesting-09-dzfdx [10.121.221.76]
  CPUs: 2 - Mem: 512 MB (336.3 MB) - Swap: 0 (0)
Feb-05 07:32:07.829 [main] DEBUG nextflow.Session - Work-dir: /storage-pvc-data/work [nfs]
Feb-05 07:32:07.830 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /proj_development/bin
Feb-05 07:32:07.944 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Feb-05 07:32:08.226 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Feb-05 07:32:08.534 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Feb-05 07:32:08.738 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 3; maxThreads: 1000
Feb-05 07:32:10.133 [main] DEBUG nextflow.Session - Session start
Feb-05 07:32:15.432 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Feb-05 07:32:16.628 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:process1` matches process process1
Feb-05 07:32:17.038 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Feb-05 07:32:17.039 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Feb-05 07:32:17.233 [main] DEBUG nextflow.executor.Executor - [warm up] executor > k8s
Feb-05 07:32:17.331 [main] DEBUG n.processor.TaskPollingMonitor - Creating task monitor for executor 'k8s' > capacity: 100; pollInterval: 5s; dumpInterval: 5m 
Feb-05 07:32:17.429 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: k8s)
Feb-05 07:32:17.833 [main] DEBUG nextflow.k8s.client.ConfigDiscovery - K8s config file does not exist: /root/.kube/config
Feb-05 07:32:18.128 [main] DEBUG nextflow.k8s.K8sExecutor - [K8s] config=[storageClaimName:storage-pvc-data, storageMountPath:/storage-pvc-data/work, namespace:nextflow, serviceAccount:next-dev, secrets:[secretKeyRef:db-credentials]]; API client config=ClientConfig[ server=172.20.0.1:443, namespace=nextflow, serviceAccount=next-dev, token=eyJhbGciOiJ.., sslCert=LS0tLS1CRUd.., clientCert=-, clientKey=-, verifySsl=false, fromFile=true, httpReadTimeout=null, httpConnectTimeout=null, maxErrorRetry=4 ]
Feb-05 07:32:19.631 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:process1_combination` matches process process1_combination
Feb-05 07:32:19.636 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Feb-05 07:32:19.636 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Feb-05 07:32:19.830 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:process2` matches process process2
Feb-05 07:32:19.942 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Feb-05 07:32:20.027 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Feb-05 07:32:20.141 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:process3` matches process process3
Feb-05 07:32:20.142 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: k8s
Feb-05 07:32:20.229 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'k8s'
Feb-05 07:32:20.336 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: process3, process1, process1_combination, process2
Feb-05 07:32:20.429 [main] DEBUG nextflow.Session - Igniting dataflow network (4)
Feb-05 07:32:20.430 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > process1
Feb-05 07:32:20.433 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > process1_combination
Feb-05 07:32:20.435 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > process2
Feb-05 07:32:20.435 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > process3
Feb-05 07:32:20.436 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_84c877d996ab2b89: /proj_development/exec-pipeline.nf
Feb-05 07:32:20.436 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination 
Feb-05 07:32:20.436 [main] DEBUG nextflow.Session - Session await
Feb-05 07:32:28.133 [Task submitter] INFO  nextflow.Session - [06/dcd3e7] Submitted process > process1
Feb-05 07:32:28.733 [Task submitter] INFO  nextflow.Session - [1f/874414] Submitted process > process1_combination
Feb-05 07:34:57.761 [Task monitor] DEBUG nextflow.k8s.K8sTaskHandler - [K8s] Cannot read exitstatus for task: `process1` | /storage-pvc-data/work/06/dcd3e7dd3afdafe0ff7fd2b3a02d51/.exitcode
Feb-05 07:34:57.956 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: process1; status: COMPLETED; exit: -; error: -; workDir: /storage-pvc-data/work/06/dcd3e7dd3afdafe0ff7fd2b3a02d51]
Feb-05 07:34:58.031 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=process1; work-dir=/storage-pvc-data/work/06/dcd3e7dd3afdafe0ff7fd2b3a02d51
  error [nextflow.exception.ProcessFailedException]: Process `process1` terminated for an unknown reason -- Likely it has been terminated by the external system
Feb-05 07:34:58.144 [Task monitor] DEBUG nextflow.processor.TaskRun - Unable to dump output of process 'null' -- Cause: java.nio.file.NoSuchFileException: /storage-pvc-data/work/06/dcd3e7dd3afdafe0ff7fd2b3a02d51/.command.out
Feb-05 07:34:58.227 [Task monitor] DEBUG nextflow.processor.TaskRun - Unable to dump error of process 'null' -- Cause: java.nio.file.NoSuchFileException: /storage-pvc-data/work/06/dcd3e7dd3afdafe0ff7fd2b3a02d51/.command.err
Feb-05 07:34:58.531 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'process1'

Caused by:
  Process `process1` terminated for an unknown reason -- Likely it has been terminated by the external system

Command executed:

< masking the command due to restrictions >

Command exit status:
  -

Command output:
  (empty)

Command wrapper:
  Unknown option: -ue -- Check the available commands and options and syntax with 'help'

Work dir:
  /storage-pvc-data/work/06/dcd3e7dd3afdafe0ff7fd2b3a02d51

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
Feb-05 07:34:58.828 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process `process1` terminated for an unknown reason -- Likely it has been terminated by the external system
Feb-05 07:34:59.631 [Task monitor] DEBUG nextflow.processor.TaskRun - Unable to dump error of process 'null' -- Cause: java.nio.file.NoSuchFileException: /storage-pvc-data/work/06/dcd3e7dd3afdafe0ff7fd2b3a02d51/.command.err
Feb-05 07:34:59.632 [Task monitor] DEBUG nextflow.processor.TaskRun - Unable to dump output of process 'null' -- Cause: java.nio.file.NoSuchFileException: /storage-pvc-data/work/06/dcd3e7dd3afdafe0ff7fd2b3a02d51/.command.out
Feb-05 07:34:59.639 [Task monitor] INFO  nextflow.script.BaseScript - Error: check logs for detailed error
Feb-05 07:34:59.730 [Task monitor] INFO  nextflow.script.BaseScript - Error: Pipeline execution stopped with the following message: null
Feb-05 07:34:59.932 [main] DEBUG nextflow.Session - Session await > all processes finished
Feb-05 07:34:59.932 [main] DEBUG nextflow.Session - Session await > all barriers passed
Feb-05 07:34:59.935 [main] DEBUG nextflow.processor.TaskRun - Unable to dump error of process 'null' -- Cause: java.nio.file.NoSuchFileException: /storage-pvc-data/work/06/dcd3e7dd3afdafe0ff7fd2b3a02d51/.command.err
Feb-05 07:34:59.936 [main] DEBUG nextflow.processor.TaskRun - Unable to dump output of process 'null' -- Cause: java.nio.file.NoSuchFileException: /storage-pvc-data/work/06/dcd3e7dd3afdafe0ff7fd2b3a02d51/.command.out
Feb-05 07:34:59.939 [main] INFO  nextflow.script.BaseScript - Pipelines completed
Feb-05 07:35:00.168 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: k8s) - terminating tasks monitor poll loop
Feb-05 07:35:00.332 [main] WARN  n.processor.TaskPollingMonitor - Killing running tasks (1)
Feb-05 07:35:00.734 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=1; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=4s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=2; peakCpus=4; peakMemory=16 GB; ]
Feb-05 07:35:01.039 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Feb-05 07:35:01.333 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

Environment

• Nextflow version: [23.10.1]
• Java version: [11]
• Operating system: [Linux]
• Bash version: (5.1.16(1)-release)

Additional context

Setup of nextflow execution in K8s is working fine with 21.04.1, while the same is failing in 23.10.1, attching image below on container command and arguments respectively from k8s dashboard. There seems to be a difference in command being sent to POD.

[raghuram-venugopal]

Hi Team, please could you help on this issue.