Error: no space left on device (Google Cloud Life Science API with docker)

[abonomi-evqlv]

Bug report

I am running Nextflow on google cloud, with a docker called Abstar. The input file is rather large (3.6 GB) and once the VM instance has started, I shortly get an error that points to a "no space left on device" error. This happens once the VM has spun up correctly.

Expected behavior

The expected behaviour is that nextflow spins up a VM instance on gcloud, gets the input from a Cloud Storage bucket, and runs the abstar command on the input file to completion. The output should then be saved in Cloud Storage bucket. I have tried this with a smaller input file (360 MB) and it runs successfully.

Steps to reproduce the problem

This is the nextflow file absatr.nf:

params.outdir = 'gs://nextflow/oputput/'
data_ch = Channel.fromPath('gs://nextflow/input/*')
​
process ABSTAR {

  publishDir "$params.outdir/", pattern: 'abstar_output/*.json'
  input:
    file part from data_ch
  output:
    file 'abstar_output/*.json' into abstar_json
    file 'abstar_output/*.log' into abstar_log
  script:
  """
  abstar -i ${part} -t /temp/ -o abstar_output;
  """
}

And this is the config file nextflow.config:

process.cpus = 96
process.memory = '100 GB'
process.disk = '2000 GB'
process.executor = 'google-lifesciences'
process.container = 'briney/abstar'
google.zone  = 'us-central1-a'
google.project = 'structure'
google.lifeSciences.debug = true

And here is the CLI:

./nextflow run Desktop/abstar_cloud.nf -with-docker briney/abstar -work-dir gs://nextflow/

Program output

.nextflow.log file:

Apr-07 13:14:18.929 [main] DEBUG nextflow.cli.Launcher - $> nextflow run Desktop/abstar_cloud.nf -with-docker briney/abstar -work-dir 'gs://antibody-sequence-compendiums/synthetic_compendium/abstar_output/'
Apr-07 13:14:19.028 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 21.10.6
Apr-07 13:14:19.062 [main] INFO  nextflow.cli.CmdRun - Launching `Desktop/abstar_cloud.nf` [amazing_agnesi] - revision: 8a8fc886b2
Apr-07 13:14:19.078 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /Users/allegra/nextflow.config
Apr-07 13:14:19.079 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /Users/allegra/nextflow.config
Apr-07 13:14:19.107 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Apr-07 13:14:19.732 [main] DEBUG nextflow.config.ConfigBuilder - Enabling execution in Docker container as requested by cli option `-with-docker briney/abstar`
Apr-07 13:14:19.772 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; plugins-dir=/Users/allegra/.nextflow/plugins
Apr-07 13:14:19.774 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[[email protected]]
Apr-07 13:14:19.791 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Apr-07 13:14:19.793 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Apr-07 13:14:19.798 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Apr-07 13:14:19.812 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Apr-07 13:14:19.812 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-google version: 1.1.3
Apr-07 13:14:19.828 [main] INFO  org.pf4j.AbstractPluginManager - Plugin '[email protected]' resolved
Apr-07 13:14:19.828 [main] INFO  org.pf4j.AbstractPluginManager - Start plugin '[email protected]'
Apr-07 13:14:19.895 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started [email protected]
Apr-07 13:14:19.983 [main] DEBUG nextflow.Session - Session uuid: b032eb20-a230-4882-aff8-bceb118a8ab0
Apr-07 13:14:19.983 [main] DEBUG nextflow.Session - Run name: amazing_agnesi
Apr-07 13:14:19.984 [main] DEBUG nextflow.Session - Executor pool size: 8
Apr-07 13:14:20.100 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 21.10.6 build 5660
  Created: 21-12-2021 16:55 UTC (11:55 GMT-04:00)
  System: Mac OS X 11.1
  Runtime: Groovy 3.0.9 on Java HotSpot(TM) 64-Bit Server VM 16.0.1+9-24
  Encoding: UTF-8 (UTF-8)
  Process: [email protected] [127.0.0.1]
  CPUs: 8 - Mem: 32 GB (40 MB) - Swap: 2 GB (371 MB)
Apr-07 13:14:20.131 [main] DEBUG nextflow.Session - Work-dir: gs://antibody-sequence-compendiums/synthetic_compendium/abstar_output/ [Mac OS X]
Apr-07 13:14:20.131 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /Users/allegra/Desktop/bin
Apr-07 13:14:20.169 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[GoogleLifeSciencesExecutor]
Apr-07 13:14:20.182 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Apr-07 13:14:20.220 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 9; maxThreads: 1000
Apr-07 13:14:20.320 [main] DEBUG nextflow.Session - Session start invoked
Apr-07 13:14:20.599 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Apr-07 13:14:20.643 [PathVisitor-1] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: /synthetic_compendium/abstar_output/input/; pattern: *; options: [:]
Apr-07 13:14:20.707 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: google-lifesciences
Apr-07 13:14:20.707 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'google-lifesciences'
Apr-07 13:14:20.709 [main] DEBUG nextflow.executor.Executor - [warm up] executor > google-lifesciences
Apr-07 13:14:20.725 [main] DEBUG n.processor.TaskPollingMonitor - Creating task monitor for executor 'google-lifesciences' > capacity: 1000; pollInterval: 10s; dumpInterval: 5m 
Apr-07 13:14:20.740 [main] DEBUG n.c.g.l.GoogleLifeSciencesExecutor - Google Life Science config=GoogleLifeSciencesConfig(project:structure-286115, zones:[us-central1-a], regions:[], preemptible:false, remoteBinDir:null, location:us-central1, disableBinDir:false, bootDiskSize:null, cpuPlatform:null, sshDaemon:false, sshImage:gcr.io/cloud-genomics-pipelines/tools, debugMode:1, copyImage:google/cloud-sdk:slim, usePrivateAddress:false, enableRequesterPaysBuckets:false, network:null, subnetwork:null, serviceAccountEmail:null, parallelThreadCount:1, downloadMaxComponents:8, keepAliveOnFailure:false, maxParallelTransfers:4, maxTransferAttempts:1, delayBetweenAttempts:10s)
Apr-07 13:14:21.079 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 9; maxThreads: 1000
Apr-07 13:14:21.174 [main] DEBUG nextflow.Session - Workflow process names [dsl1]: ABSTAR
Apr-07 13:14:21.174 [main] DEBUG nextflow.script.ScriptRunner - > Await termination 
Apr-07 13:14:21.174 [main] DEBUG nextflow.Session - Session await
Apr-07 13:14:23.521 [Task submitter] DEBUG n.c.g.l.GoogleLifeSciencesTaskHandler - [GLS] Task submitted > ABSTAR (1) - Pipeline Id: 4161915762647648455
Apr-07 13:14:23.521 [Task submitter] INFO  nextflow.Session - [18/9a35ce] Submitted process > ABSTAR (1)
Apr-07 13:18:01.041 [Task monitor] DEBUG n.c.g.l.GoogleLifeSciencesTaskHandler - [GLS] Task complete > ABSTAR (1) - Start Time: 2022-04-07T17:14:35.557919591Z - End Time: 2022-04-07T17:17:51.997214548Z
Apr-07 13:18:01.216 [Task monitor] DEBUG n.c.g.l.GoogleLifeSciencesTaskHandler - [GLS] Cannot read exitstatus for task: `ABSTAR (1)` | gs://antibody-sequence-compendiums/synthetic_compendium/abstar_output/18/9a35cef6e19af76bdc5ac1bdb84e42/.exitcode
Apr-07 13:18:01.217 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: ABSTAR (1); status: COMPLETED; exit: 9; error: -; workDir: gs://antibody-sequence-compendiums/synthetic_compendium/abstar_output/18/9a35cef6e19af76bdc5ac1bdb84e42]
Apr-07 13:18:01.294 [Task monitor] DEBUG nextflow.processor.TaskRun - Unable to dump output of process 'null' -- Cause: java.nio.file.NoSuchFileException: gs://antibody-sequence-compendiums/synthetic_compendium/abstar_output/18/9a35cef6e19af76bdc5ac1bdb84e42/.command.out
Apr-07 13:18:01.297 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'ABSTAR (1)'

Caused by:
  Process `ABSTAR (1)` terminated with an error exit status (9)

Command executed:

  abstar -i partaa -t /temp/ -o abstar_output;

Command exit status:
  9

Command output:
  (empty)

Command error:
  Execution failed: generic::failed_precondition: action 3: starting container: running container: running ["docker" "create" "-i" "--env-file" "/tmp/environment1703763328" "--name" "nf-189a35cef6e19af76bdc5ac1bdb84e42-unstage" "--network" "pipelines" "--mount" "type=bind,source=/mnt/disks/nf-pipeline-work,target=/synthetic_compendium/abstar_output/18/9a35cef6e19af76bdc5ac1bdb84e42,bind-propagation=rslave" "--mount" "type=bind,source=/var/lib/pipelines/google,target=/google,readonly,bind-propagation=rslave" "--shm-size" "70352m" "--oom-score-adj" "1000" "--log-driver" "none" "google/cloud-sdk:slim" "bash" "-c" "set -x; trap 'err=$?; exec 1>&2; gsutil -m -q cp -R /synthetic_compendium/abstar_output/18/9a35cef6e19af76bdc5ac1bdb84e42/.command.log gs://antibody-sequence-compendiums/synthetic_compendium/abstar_output/18/9a35cef6e19af76bdc5ac1bdb84e42/.command.log || true; [[ $err -gt 0 || $GOOGLE_LAST_EXIT_STATUS -gt 0 || $NXF_DEBUG -gt 0 ]] && { ls -lah /synthetic_compendium/abstar_output/18/9a35cef6e19af76bdc5ac1bdb84e42 || true; gsutil -m -q cp -R /google/ gs://antibody-sequence-compendiums/synthetic_compendium/abstar_output/18/9a35cef6e19af76bdc5ac1bdb84e42; } || rm -rf /synthetic_compendium/abstar_output/18/9a35cef6e19af76bdc5ac1bdb84e42; exit $err' EXIT; { cd /synthetic_compendium/abstar_output/18/9a35cef6e19af76bdc5ac1bdb84e42; bash .command.run nxf_unstage; } >> /synthetic_compendium/abstar_output/18/9a35cef6e19af76bdc5ac1bdb84e42/.command.log 2>&1"]: exit status 1 (standard error: "Error response from daemon: mkdir /var/lib/docker/overlay2/a451787904acca5ddbdc9b65a867561ea179cf6d75f61da734c5953332fc06fe-init/diff: no space left on device\n")

Work dir:
  gs://antibody-sequence-compendiums/synthetic_compendium/abstar_output/18/9a35cef6e19af76bdc5ac1bdb84e42

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
Apr-07 13:18:01.305 [main] DEBUG nextflow.Session - Session await > all process finished
Apr-07 13:18:01.360 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process `ABSTAR (1)` terminated with an error exit status (9)
Apr-07 13:18:01.381 [main] DEBUG nextflow.Session - Session await > all barriers passed
Apr-07 13:18:01.395 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=3h 44m 13s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=64; peakMemory=70 GB; ]
Apr-07 13:18:01.540 [main] DEBUG nextflow.CacheDB - Closing CacheDB done
Apr-07 13:18:01.541 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin '[email protected]'
Apr-07 13:18:01.541 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-google
Apr-07 13:18:01.554 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

Additional context

I believe this may be coming from no more space on the docker image. However, I do not understand why it would run out of space, given that the boot disk is extremely large. I see online some people recommending to docker system prune however, how can I do this, given that it is nextflow that is spinning up these VMs?

Thanks for the help!

[bentsherman]

Hi @abonomi-evqlv , you need to use the google.lifeSciences.bootDiskSize option to increase the boot disk size. I believe it is 10 GB by default. The disk directive doesn't change the boot disk, but instead provisions a second volume.

I've had this error before because my docker image was too large for the boot disk.

Additionally, consider using the new Google Batch executor with the latest Nextflow edge release. Not directly related to your issue, but it should give better performance in general.