Pipeline run with docker breaks with last NF version

[dgarrimar]

Bug report

Expected behavior and actual behavior

I have a CI Github action that runs a Nextflow pipeline on a test dataset, after installing Nextflow itself. It runs the pipeline within a Docker container via the -with-docker option. After the last Nextflow update (20.10.0), without changing the pipeline, I found that the CI failed. After a couple of checks, it seems to work when using a previous version of Nextflow, e.g. NXF_VER=20.04.0. The same behaviour is observed when running the pipeline directly on Mac OS and Linux. It also works independently of the Nextflow version if using -with-singularity instead, so the problem seems related to the link Nextflow-Docker.

Steps to reproduce the problem

You can try running nextflow run guigolab/sqtlseeker2-nf -with-docker.
You can also fork/clone the repo, make some edit in the README, push and check the outcome of the action.

Program output

It complains about a missing output file, but I believe it is related to the fact that it can't run within the container.

Caused by:
  Missing output file(s) `tre.df.RData` expected by process `prepare (tissue1)`

Command executed:

  prepare_trexp.R --group "tissue1" -t texpression.tsv.gz -m metadata.tsv --gene_location genes.bed --output_tre tre.df.RData --output_gene genes.ss.bed --output_cov covariates.df.RData

Command exit status:
  0

Command output:
  (empty)

[pditommaso]

Diego! I'm sure you can do more to troubleshoot this problem 😄

[dgarrimar]

Courtesy of a beloved common friend ;)

diff between the .command.run of NF 22.04.0 and NF 22.10.0
first (c8/223da) is 22.04.0

in ~/tmp/test-sqtlseeker took 47s 
❯ diff work/c8/223da30967c74ded694978018eb036/.command.run work/bd/e13fcd6204dfd465e6c0d191230a4c/.command.run 
123c123
<     /bin/bash -ue /Users/epalumbo/tmp/test-sqtlseeker/work/c8/223da30967c74ded694978018eb036/.command.sh
---
>     /bin/bash -ue /Users/epalumbo/tmp/test-sqtlseeker/work/bd/e13fcd6204dfd465e6c0d191230a4c/.command.sh
148c148
<     /bin/bash -ue /Users/epalumbo/tmp/test-sqtlseeker/work/c8/223da30967c74ded694978018eb036/.command.sh &
---
>     /bin/bash -ue /Users/epalumbo/tmp/test-sqtlseeker/work/bd/e13fcd6204dfd465e6c0d191230a4c/.command.sh &
197c197
< export PATH="/Users/epalumbo/.nextflow/assets/guigolab/sqtlseeker2-nf/bin:\$PATH"
---
> export PATH="\$PATH:/Users/epalumbo/.nextflow/assets/guigolab/sqtlseeker2-nf/bin"
236a237
>     mkdir -p "$base"
264c265
<     printf $exit_status > /Users/epalumbo/tmp/test-sqtlseeker/work/c8/223da30967c74ded694978018eb036/.exitcode
---
>     printf $exit_status > /Users/epalumbo/tmp/test-sqtlseeker/work/bd/e13fcd6204dfd465e6c0d191230a4c/.exitcode
279c280
<     docker run -i --cpus 1.0 -e "NXF_DEBUG=${NXF_DEBUG:=0}" -v /Users/epalumbo:/Users/epalumbo -v "$PWD":"$PWD" -w "$PWD" --entrypoint /bin/bash -u $(id -u) --name $NXF_BOXID ghcr.io/guigolab/sqtlseeker2-nf@sha256:482776f5efaf7ae02d550b9ee81fc316ce545a55aaec8e9c44cae0985e0052f3 -c "eval $(nxf_container_env); /bin/bash /Users/epalumbo/tmp/test-sqtlseeker/work/c8/223da30967c74ded694978018eb036/.command.run nxf_trace"
---
>     docker run -i --cpus 1.0 -e "NXF_DEBUG=${NXF_DEBUG:=0}" -v /Users/epalumbo:/Users/epalumbo -w "$PWD" -u $(id -u) --name $NXF_BOXID ghcr.io/guigolab/sqtlseeker2-nf@sha256:482776f5efaf7ae02d550b9ee81fc316ce545a55aaec8e9c44cae0985e0052f3 /bin/bash -c "eval $(nxf_container_env); /bin/bash /Users/epalumbo/tmp/test-sqtlseeker/work/bd/e13fcd6204dfd465e6c0d191230a4c/.command.run nxf_trace"
303c304
<     touch /Users/epalumbo/tmp/test-sqtlseeker/work/c8/223da30967c74ded694978018eb036/.command.begin
---
>     touch /Users/epalumbo/tmp/test-sqtlseeker/work/bd/e13fcd6204dfd465e6c0d191230a4c/.command.begin
306c307
<     [[ $NXF_SCRATCH ]] && echo "nxf-scratch-dir $HOSTNAME:$NXF_SCRATCH" && cd $NXF_SCRATCH
---
>     [[ $NXF_SCRATCH ]] && cd $NXF_SCRATCH 

entrypoint?

[pditommaso]

Mucb better. Nextflow does not override anymore the container entrypoint as of version 22.08

https://github.com/nextflow-io/nextflow/blob/master/changelog.txt#L169

You can use the following env variable to preserve the old behaviour, but at some point it will go away

export NXF_CONTAINER_ENTRYPOINT_OVERRIDE=true