When sub-command of a pipe fails, jobs shows as COMPLETED in SLURM and Nextflow continues

[leonorpalmeira]

Bug report

When sub-command of a pipe fails, jobs shows as COMPLETED in SLURM and Nextflow continues.

Expected behavior and actual behavior

When the following seqtk invocation fails due to lack of allocated RAM in SLURM, the job shows as COMPLETED in SLURM and Nextflow continues execution (.exitcode is present with value 0):

seqtk sample input.fastq.gz ${value} | gzip - > output.fastq.gz

I was expecting this job to fail and to lead to Nextflow failing as well.

Steps to reproduce the problem

The current .command.sh is the following and incorporates a pipe. seqtk will fail whenever RAM allocation is too scarce:

#!/bin/bash -ue
value=15000000
seqtk sample input.fastq.gz ${value} | gzip - > output.fastq.gz

If I add set -o pipefail to this script and re-launch .command.run, then the appropriate behavior is rescued (SLURM shows a FAIL error message and Nextflow fails) and when seqtk fails due to memory allocation, this is now properly seen by Nextflow:

#!/bin/bash -ue
set -o pipefail
value=15000000
seqtk sample input.fastq.gz ${value} | gzip - > output.fastq.gz

Do you think it would be possible to add -o to .command.sh as it is done for -e and -u? Or would this interfere with other use cases?

Environment

• Nextflow version: 22.10.3
• Java version: 8.0.111
• Operating system: Scientific Linux
• Bash version: 4.2.46(2)-release in SLURM, 4.4.20(1)-release inside Singularity container

[mribeirodantas]

You can change the shebang of the .command.sh with:

process.shell = ['/bin/bash', '-ue', .. ]

For example, add the following to your nextflow.config:

process.shell = ['/bin/bash', '-ueo','pipefail' ]

Can you try that and tell me if it works? 😄

[leonorpalmeira]

Yes, this works fine and is cleaner than my current workaround. Thank you!

[mribeirodantas]

Yay! 🥳