.exitcode not correctly produced with SGE.

[snalty]

Bug report

Expected behavior and actual behavior

A pipeline written by a colleague of mine fails to run when using SGE as an executor. The local executor works fine.

The pipeline hangs, however I can see the output files for each process are correctly produced. I believe this hang is due to the .exitcode file not being correctly produced.

Steps to reproduce the problem

Run the pipeline with the SGE executor.

Program output

nextflow log file can be found at the following gist:

https://gist.github.com/snalty/1a02fc0a8b0b3cf0f814d16f604fb743

Further to the nextflow log, the SGE job log contains the following:

mkfifo: error while loading shared libraries: libdl.so.2: failed to map segment from shared object
mkfifo: error while loading shared libraries: libdl.so.2: failed to map segment from shared object
/var/spool/gridengine/execd/brain/job_scripts/28254: line 124: /dev/shm/nxf.4TizkX8IAL/.command.out: No such file or directory
/var/spool/gridengine/execd/brain/job_scripts/28254: line 126: /dev/shm/nxf.4TizkX8IAL/.command.err: No such file or directory
/var/spool/gridengine/execd/brain/job_scripts/28254: xrealloc: cannot allocate 33554432 bytes

Environment

• Nextflow version: nextflow version 22.04.5.5708
• Java version: openjdk 18.0.2.1 2022-08-18
• Operating system: Ubuntu 18.04.3 LTS
• Bash version: GNU bash, version 4.4.20(1)-release

Additional context

Thanks for the help.

[snalty]

Interestingly, when I try and run nextflow with LD_DEBUG=all, I get the following error message:

     38777:
     38777:     calling fini: /lib/x86_64-linux-gnu/libtinfo.so.5 [0]
     38777:
     38777:
     38777:     calling fini: /lib/x86_64-linux-gnu/libdl.so.2 [0]
     38777:
ERROR: Cannot find Java or it's a wrong version -- please make sure that Java 8 or later (up to 18) is installed
NOTE: Nextflow is trying to use the Java VM defined by the following environment variables:
 JAVA_CMD: /home/pipeline/.local/jdk-18.0.2.1/bin/java
 NXF_JAVA_HOME: /home/pipeline/.local/jdk-18.0.2.1

     38732:
     38732:     calling fini: bash [0]
     38732:
     38732:
     38732:     calling fini: /lib/x86_64-linux-gnu/libtinfo.so.5 [0]

although that might be a red herring perhaps?

[snalty]

Furthermore, here is the .command.log output when adding set -x to the .command.run file:

mkfifo: error while loading shared libraries: libdl.so.2: failed to map segment from shared object
mkfifo: error while loading shared libraries: libdl.so.2: failed to map segment from shared object
/var/spool/gridengine/execd/brain/job_scripts/28429: line 124: /dev/shm/nxf.v8cbbpzHBz/.command.out: No such file or directory
/var/spool/gridengine/execd/brain/job_scripts/28429: line 126: /dev/shm/nxf.v8cbbpzHBz/.command.err: No such file or directory
/var/spool/gridengine/execd/brain/job_scripts/28429: xrealloc: cannot allocate 33554432 bytes
+ NXF_DEBUG=0
+ [[ 0 > 1 ]]
+ NXF_ENTRY=nxf_main
+ nxf_main
+ trap on_exit EXIT
+ trap on_term TERM INT USR2
+ trap '' USR1
+ [[ -n '' ]]
+ NXF_SCRATCH=
+ [[ 0 > 0 ]]
+ touch /home/gensan/dev/MSI_pipeline_dev/work/00/4b04d6d3ca9bdce01ead37bd4d4a55/.command.begin
+ set +u
+ set -u
+ export PATH=/home/gensan/dev/MSI_pipeline_dev/bin:/tmp/28453.1.sdgs-clinical:/usr/local/bin:/bin:/usr/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin
+ PATH=/home/gensan/dev/MSI_pipeline_dev/bin:/tmp/28453.1.sdgs-clinical:/usr/local/bin:/bin:/usr/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin
+ [[ -n '' ]]
+ nxf_stage
+ true
+ set +e
++ set +u
++ nxf_mktemp /dev/shm
+ ctmp=/dev/shm/nxf.sTD2NOmWwJ
+ local cout=/dev/shm/nxf.sTD2NOmWwJ/.command.out
+ mkfifo /dev/shm/nxf.sTD2NOmWwJ/.command.out
mkfifo: error while loading shared libraries: libdl.so.2: failed to map segment from shared object
+ local cerr=/dev/shm/nxf.sTD2NOmWwJ/.command.err
+ mkfifo /dev/shm/nxf.sTD2NOmWwJ/.command.err
mkfifo: error while loading shared libraries: libdl.so.2: failed to map segment from shared object
+ tee1=44502
+ tee .command.out
+ tee2=44503
/var/spool/gridengine/execd/brain/job_scripts/28453: line 125: /dev/shm/nxf.sTD2NOmWwJ/.command.out: No such file or directory
+ tee .command.err
+ pid=44504
+ wait 44504
/var/spool/gridengine/execd/brain/job_scripts/28453: line 127: /dev/shm/nxf.sTD2NOmWwJ/.command.err: No such file or directory
+ wait 44502 44503
/var/spool/gridengine/execd/brain/job_scripts/28453: xrealloc: cannot allocate 33554432 bytes

Running the commands that fail
mkfifo /dev/shm/nxf.sTD2NOmWwJ/.command.out

From the shell works perfectly fine, and I can see that the file that apparently doesn't exist:

/var/spool/gridengine/execd/brain/job_scripts/28453: line 125: /dev/shm/nxf.sTD2NOmWwJ/.command.out: No such file or directory

does actually exist on the file system. This seems to me like perhaps a permissions issue.

[snalty]

Ugh I'm so dumb, it genuinely was looking me right in the face:

/var/spool/gridengine/execd/brain/job_scripts/28254: xrealloc: cannot allocate 33554432 bytes

The job simply needed more memory!

process.clusterOptions = '-l h_vmem=100M' in the config fixed it.