Nextflow process docker commands do not work on bitbucket pipelines cloud platform, all volumes must be mounted under $BITBUCKET_CLONE_DIR and its subdirectories.

[mbeavitt]

Bug report

Expected behavior and actual behavior

In developing a CI/CD development strategy for a nextflow pipeline using the nf-core template using bitbucket pipelines, I was expecting to be able to mount and run containers correctly straight away. Unfortunately, in the .command.run file it is clear that nextflow initialises containers by mounting directories from /etc/passwd and /etc/shadow, which are not permitted on the bitbucket cloud platform.

Steps to reproduce the problem

To replicate the problem, you can try to run any test pipeline, e.g. nf-core, on the bitbucket pipelines platform. e.g. using the command:

nextflow run main.nf -profile test,docker --outdir test_output -w $BITBUCKET_CLONE_DIR/work

in the bitbucket-pipelines.yml test script section.

Program output

executor >  local (1)

[63/8ef24d] process > NFCORE_BHASM:BHASM:INPUT_CH... [100%] 1 of 1, failed: 1 ✘

[-        ] process > NFCORE_BHASM:BHASM:FASTQC      -

[-        ] process > NFCORE_BHASM:BHASM:CUSTOM_D... -

[-        ] process > NFCORE_BHASM:BHASM:MULTIQC     [  0%] 0 of 1

Execution cancelled -- Finishing pending tasks before exit

-[nf-core/bhasm] Pipeline completed with errors-

ERROR ~ Error executing process > 'NFCORE_BHASM:BHASM:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_test_illumina_amplicon.csv)'

Caused by:

  Process `NFCORE_BHASM:BHASM:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_test_illumina_amplicon.csv)` terminated with an error exit status (125)

Command executed:

  check_samplesheet.py \

      samplesheet_test_illumina_amplicon.csv \

      samplesheet.valid.csv

  

  cat <<-END_VERSIONS > versions.yml

  "NFCORE_BHASM:BHASM:INPUT_CHECK:SAMPLESHEET_CHECK":

      python: $(python --version | sed 's/Python //g')

  END_VERSIONS

Command exit status:

  125

Command output:

  (empty)

Command error:

  docker: Error response from daemon: authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories.

  See 'docker run --help'.

Work dir:

  /opt/atlassian/pipelines/agent/build/work/63/8ef24d42e52dd639333932350f91ec

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details

command failed

Environment

• Nextflow version: 23.04.3
• Java version:

root@4c591253b44d:/usr/src/nf_core# java -version
openjdk version "17.0.8" 2023-07-18
OpenJDK Runtime Environment (build 17.0.8+7-Debian-1deb12u1)
OpenJDK 64-Bit Server VM (build 17.0.8+7-Debian-1deb12u1, mixed mode, sharing)

• Operating system: Debian GNU/Linux 12 (bookworm), from nfcore/tools image
• Bash version: (use the command $SHELL --version)

GNU bash, version 5.2.15(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Additional context

I guess the solution here is to route the docker volume paths to a location that is allowed on bitbucket pipeline.
From the error, -v only supports $BITBUCKET_CLONE_DIR and its subdirectories on the platform.
This is the problematic docker command that nextflow is running:

nxf_launch() {
    docker run -i --cpu-shares 1024 --memory 6144m -e "NXF_DEBUG=${NXF_DEBUG:=0}" -u $(id -u) -e "HOME=${HOME}" -v /etc/passwd:/etc/passwd:ro -v /etc/shadow:/etc/shadow:ro -v /etc/group:/etc/group:ro -v $HOME:$HOME -v /opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build -w "$PWD" --name $NXF_BOXID quay.io/biocontainers/python:3.8.3 /bin/bash -c "eval $(nxf_container_env); /bin/bash /opt/atlassian/pipelines/agent/build/work/63/8ef24d42e52dd639333932350f91ec/.command.run nxf_trace"
}

in particular:

• -v /etc/shadow:/etc/shadow:ro
• -v /etc/passwd:/etc/passwd:ro
• -v /etc/group:/etc/group:ro

I've attached my bitbucket yaml file, which runs various testing scripts, along with the associated pipeline output file containing the results of the commands. I've also included the .nextflow.log file, and the .command.run file from nextflow.

pipelineLog-{baed7c6b-0968-489e-9d83-2b5a07688b70}.txt
nextflow.txt
command.txt
bitbucket-pipelines.txt

[pditommaso]

This only happens when the (UNDOCUMENTED) setting userEmulation is added in the config.

I believe this is a problem with nf-core pipeline. @drpatelh @ewels can this setting please removed from nf-core config

[ewels]

Aye aye cap'n: nf-core/tools#2580

[pditommaso]

Aye aye cap'n

[pditommaso]

Related to this #4596

[mbeavitt]

I just tried to run the same test with the command:

nextflow run main.nf -profile test,docker --outdir test_output -w $BITBUCKET_CLONE_DIR/work --docker.userEmulation false

But it still tried to mount volumes in /etc/.

I can confirm that the issue was solved however by modifying the nextflow.config file and changing the relevant setting.

Is there another way to change this behaviour temporarily, in the run command? Am I doing something wrong?
I'd like user emulation with normal runs, but test runs on the cloud pipeline can be run without user emulation.

[pditommaso]

Don't think --docker.userEmulation is a valid parameter. Leave to nf-core folks to reply

[mbeavitt]

Ok, thanks. In the meantime I've just created a clone config file for the test runs with that parameter set to false, and everything is working as desired now.

Appreciate the help, thank you so much

[ewels]

Single hyphens are core options, double are workflow parameters. So here you're setting params.docker.userEmulation, which is not used by the pipeline and ignored. You'd typically want -docker.userEmulation for a core Nextflow flag, but there is no such Nextflow CLI option. You need to put this one in a confif file and point to that.

No need to clone the pipeline config, you can create a one-line config file and use that. Nextflow configuration can be loaded from many places, see the docs.