Docker permission error using Slurm on HPC with Nextflow

[jdlamstein]

Bug report

Similar issues include:
#2583
https://forums.docker.com/t/got-permission-denied-while-trying-to-connect-to-the-docker-daemon-socket/113292

Expected behavior and actual behavior

I'm submitting a test workflow into the HPC with Nextflow. The test runs a hello world bash script. When I run this, I get a docker socket permission error:

docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create?name=nxf-vfPUNFEXBmP4OS8RLgRNOlTh": dial unix /var/run/docker.sock: connect: permission denied.

From what I can see, the gid of /var/run/docker.sock is included in id --groups.

Steps to reproduce the problem

I run

nextflow run simpletest.nf

My simpletest.nf file is:

#!/usr/bin/env nextflow
/*
 * pipeline input parameters
 */

params.text = 'Hello World!'


text_ch = Channel.of(params.text)



params.outdir = "results"

log.info """\
    B A S H  H E L L O  W O R L D !
    ===================================
    bash input : ${params.text}
    """
    .stripIndent()



process BASHEX {
    tag "Bash Script Test"

    input:
    val x

    output:
    val true
    
    script:
    """
    test.sh
    """
}

workflow {
    BASHEX(text_ch)
    bashresults_ch = BASHEX.out
}

workflow.onComplete {
    log.info ( workflow.success ? "\nDone! Open the following report in your browser --> $params.outdir/pyexample_report.html\n" : "Oops .. something went wrong" )
}

And my nextflow.config file is:

process.container = 'jdlamstein/datastudy'
docker.runOptions = '-u $(id -u):$(id -g)'
docker.enabled = true
process {
    executor = 'slurm'
    queue = 'work'
    memory = '10 GB'
    time = '30 min'
    cpus = 4
}

Program output

Below is the .nextflow.log

Jan-09 14:12:00.547 [main] DEBUG nextflow.cli.Launcher - $> nextflow run simpletest.nf
Jan-09 14:12:00.654 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 23.10.0
Jan-09 14:12:00.674 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/finkbeiner/imaging/home/jlamstein/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Jan-09 14:12:00.689 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jan-09 14:12:00.691 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jan-09 14:12:00.694 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Jan-09 14:12:00.705 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jan-09 14:12:00.723 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /imaging-work/datastudy/nextflow.config
Jan-09 14:12:00.725 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file:/imaging-work/datastudy/nextflow.config
Jan-09 14:12:00.746 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Jan-09 14:12:01.478 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Jan-09 14:12:01.497 [main] INFO  nextflow.cli.CmdRun - Launching `simpletest.nf` [infallible_pare] DSL2 - revision: f10ea95ea1
Jan-09 14:12:01.498 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jan-09 14:12:01.498 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jan-09 14:12:01.509 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /finkbeiner/imaging/home/jlamstein/.nextflow/secrets/store.json
Jan-09 14:12:01.513 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@3b545206] - activable => nextflow.secret.LocalSecretsProvider@3b545206
Jan-09 14:12:01.583 [main] DEBUG nextflow.Session - Session UUID: f8b8bc40-5320-4bc8-ba45-67bd509b415e
Jan-09 14:12:01.583 [main] DEBUG nextflow.Session - Run name: infallible_pare
Jan-09 14:12:01.584 [main] DEBUG nextflow.Session - Executor pool size: 3
Jan-09 14:12:01.592 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jan-09 14:12:01.597 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=10; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jan-09 14:12:01.633 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 23.10.0 build 5889
  Created: 15-10-2023 15:07 UTC (08:07 PDT)
  System: Linux 4.18.0-372.26.1.el8_6.x86_64
  Runtime: Groovy 3.0.19 on OpenJDK 64-Bit Server VM 17.0.6+10
  Encoding: UTF-8 (UTF-8)
  Process: [email protected] [10.1.101.49]
  CPUs: 3 - Mem: 5.5 GB (1.9 GB) - Swap: 5.9 GB (5.3 GB)
Jan-09 14:12:01.672 [main] DEBUG nextflow.Session - Work-dir:/imaging-work/datastudy/work [nfs]
Jan-09 14:12:01.707 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jan-09 14:12:01.718 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jan-09 14:12:01.766 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jan-09 14:12:01.777 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 4; maxThreads: 1000
Jan-09 14:12:01.983 [main] DEBUG nextflow.Session - Session start
Jan-09 14:12:02.205 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jan-09 14:12:02.251 [main] INFO  nextflow.Nextflow - B A S H  H E L L O  W O R L D !
===================================
bash input : Hello World!

Jan-09 14:12:02.317 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jan-09 14:12:02.318 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jan-09 14:12:02.329 [main] DEBUG nextflow.executor.Executor - [warm up] executor > slurm
Jan-09 14:12:02.335 [main] DEBUG n.processor.TaskPollingMonitor - Creating task monitor for executor 'slurm' > capacity: 100; pollInterval: 5s; dumpInterval: 5m 
Jan-09 14:12:02.337 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: slurm)
Jan-09 14:12:02.344 [main] DEBUG n.executor.AbstractGridExecutor - Creating executor 'slurm' > queue-stat-interval: 1m
Jan-09 14:12:02.432 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: BASHEX
Jan-09 14:12:02.433 [main] DEBUG nextflow.Session - Igniting dataflow network (2)
Jan-09 14:12:02.433 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > BASHEX
Jan-09 14:12:02.435 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_1670075a3f17f6a4: /imaging-work/datastudy/simpletest.nf
Jan-09 14:12:02.435 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination 
Jan-09 14:12:02.435 [main] DEBUG nextflow.Session - Session await
Jan-09 14:12:02.667 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process BASHEX (Bash Script Test) > jobId: 2188; workDir:/imaging-work/datastudy/work/05/f29fcc996dd802aae2369df6eaae21
Jan-09 14:12:02.668 [Task submitter] INFO  nextflow.Session - [05/f29fcc] Submitted process > BASHEX (Bash Script Test)
Jan-09 14:12:07.382 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 2188; id: 1; name: BASHEX (Bash Script Test); status: COMPLETED; exit: 126; error: -; workDir: /imaging-work/datastudy/work/05/f29fcc996dd802aae2369df6eaae21 started: 1704838327354; exited: 2024-01-09T22:12:03.075067Z; ]
Jan-09 14:12:07.390 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=BASHEX (Bash Script Test); work-dir=/imaging-work/datastudy/work/05/f29fcc996dd802aae2369df6eaae21
  error [nextflow.exception.ProcessFailedException]: Process `BASHEX (Bash Script Test)` terminated with an error exit status (126)
Jan-09 14:12:07.444 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'BASHEX (Bash Script Test)'

Caused by:
  Process `BASHEX (Bash Script Test)` terminated with an error exit status (126)

Command executed:

  test.sh

Command exit status:
  126

Command output:
  (empty)

Command error:
  docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create?name=nxf-vfPUNFEXBmP4OS8RLgRNOlTh": dial unix /var/run/docker.sock: connect: permission denied.
  See 'docker run --help'.

Work dir:
  /imaging-work/datastudy/work/05/f29fcc996dd802aae2369df6eaae21

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
Jan-09 14:12:07.464 [main] DEBUG nextflow.Session - Session await > all processes finished
Jan-09 14:12:07.464 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process `BASHEX (Bash Script Test)` terminated with an error exit status (126)
Jan-09 14:12:07.495 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jan-09 14:12:07.496 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: slurm) - terminating tasks monitor poll loop
Jan-09 14:12:07.500 [main] INFO  nextflow.Nextflow - Oops .. something went wrong
Jan-09 14:12:07.505 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=112ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=4; peakMemory=10 GB; ]
Jan-09 14:12:07.722 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jan-09 14:12:07.754 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

Environment

• Nextflow version: 23.10.0.5889
• Java version: Temurin-17.0.6+10
• Operating system: RHEL 8
• Bash version: GNU bash, version 4.4.20(1)-release (x86_64-redhat-linux-gnu)

Additional context

[jjfarrell]

System admins do not allow docker to run on HPC systems because of security concerns. So the error is likely related to your local HPC policy rather than a bug. Instead, try switching to Singularity which does not have those Docker security concerns and is widely available on HPC systems. Here are some details for using Singularity with Nextflow (https://www.nextflow.io/docs/latest/singularity.html).

I develop my docker images on my local iMac with Docker, push the container to the Docker hub, and then pull them with Nextflow/Singularty on the HPC.