zerolog: could not write event: write /tmp/fusionfs.log175481470: no space left on device

[nick-youngblut]

Bug report

Expected behavior and actual behavior

Running Nextflow using nf-tower, Fusion, Wave, and AWS Batch.

Steps to reproduce the problem

Devcontainer:

{
    "name": "nfcore",
    "image": "nyoungb2/nf-core:2.7.2",   # nextflow 22.10.6 installed 
    "remoteUser": "gitpod",
    "customizations": {
        "vscode": {
            "settings": {
                "python.defaultInterpreterPath": "/opt/conda/bin/python",
                "python.linting.enabled": true,
                "python.linting.pylintEnabled": true,
                "python.formatting.autopep8Path": "/opt/conda/bin/autopep8",
                "python.formatting.yapfPath": "/opt/conda/bin/yapf",
                "python.linting.flake8Path": "/opt/conda/bin/flake8",
                "python.linting.pycodestylePath": "/opt/conda/bin/pycodestyle",
                "python.linting.pydocstylePath": "/opt/conda/bin/pydocstyle",
                "python.linting.pylintPath": "/opt/conda/bin/pylint"
            },
            "extensions": ["ms-python.python", "ms-python.vscode-pylance", "REditorSupport.r", "EditorConfig.EditorConfig",
                           "codezombiech.gitignore", "Gruntfuggly.todo-tree", "ms-vsliveshare.vsliveshare",
                           "nextflow.nextflow", "redhat.vscode-yaml", "streetsidesoftware.code-spell-checker"]
        }
	},
	"features": {
		"ghcr.io/devcontainers/features/docker-in-docker:2": {
            "version": "latest"
        },
        "ghcr.io/devcontainers/features/aws-cli:1": {
            "version": "latest"
        }
   }
}

Relevant config sections:

profiles {
    docker {
        docker.enabled         = true
        docker.sudo            = true
        docker.runOptions      = '-u \$(id -u):\$(id -g)'
        singularity.enabled    = false
        podman.enabled         = false
        shifter.enabled        = false
        charliecloud.enabled   = false
    } 
    aws {
        process.executor   = 'awsbatch'
        process.queue      = 'my_queue'
        tower.enabled      = true
        fusion.enabled     = true
        wave.enabled       = true
    }
}

aws {
  batch.cliPath = '/home/ec2-user/miniconda/bin/aws'
  accessKey = secrets.AWS_ACCESS_KEY_ID  
  secretKey = secrets.AWS_SECRET_ACCESS_KEY
  region = secrets.AWS_DEFAULT_REGION
}

tower {
  accessToken = secrets.TOWER_ACCESS_TOKEN
}

Program output

(Copy and paste here output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the .nextflow.log file.)

Environment

• Nextflow version: 22.10.6
• Java version: "17.0.3-internal" 2022-04-19
• Operating system: Ubuntu 20.04.5
• Bash version: GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)

Additional context

Full error:

Caused by:
  Essential container in task exited

Command executed:

  concat_tables.py -d , -p "qual-report_taxid-{taxid}.csv" "qual-report*.csv" > qual-report.csv

Command exit status:
  -

Command output:
  (empty)

Command error:
  (more omitted..)
  zerolog: could not write event: write /tmp/fusionfs.log175481470: no space left on device
  zerolog: could not write event: write /tmp/fusionfs.log175481470: no space left on device
  zerolog: could not write event: write /tmp/fusionfs.log175481470: no space left on device
  zerolog: could not write event: write /tmp/fusionfs.log175481470: no space left on device
  zerolog: could not write event: write /tmp/fusionfs.log175481470: no space left on device
 [...]

concat_tables.py is a very simple python script to concatenate small tsv/csv tables.

[pditommaso]

Are you launching this via Tower or using the nextflow CLI?

[nick-youngblut]

I'm just using nextflow run with the following:

tower.enabled      = true
fusion.enabled     = true
wave.enabled       = true

[pditommaso]

@jordeu thoughts?

[jordeu]

Fusion while the task is running is storing the logs into /tmp folder of the container and only at the end uploads it to S3 task working directory. This error means that there is no space space left on /tmp device.

Fusion uses /tmp as temporal storage a chunk cache of downloads and uploads to S3, it keeps monitoring the space left on the device and evicts chunks as fast as possible. But if there is no space even for the logs this means that the device size is very small. As explained here, the performance of /tmp device is very important to get good Fusion throughput.

How is the AWS my_queue configured? How much space is available at the container /tmp device?

[jordeu]

Notice that if you do not set process.scratch = false also Nextflow is using /tmp as scratch folder where the process is run.

[nick-youngblut]

How is the AWS my_queue configured? How much space is available at the container /tmp device?

This seems like a general challenge for reproducibility of AWS Batch issues with Nextflow, given all of the possible configurations for queues, compute environments, and AMIs. Would it be possible to add a feature to Nextflow to automatically gather relevant AWS Batch information?

I've hit the problem of filling up /tmp on various machine many times over the years, but what is especially odd about this current issue is the low amount of disk space that should be needed for these jobs. I'm just concatenating some small CSV tables using the attached script, which concats the tables and adds table columns based on metadata embedded in the file names (here, the taxid in the file names; my command: concat_tables.py -d , -p "qual-report_taxid-{taxid}.csv" "qual-report*.csv" > qual-report.csv). I don't see why this should fill up /tmp/.

I have since tried to re-run these concat_tables.py jobs via -resume, and the jobs seem to "stall" in "running" mode on AWS Batch (the jobs complete in <1 minute when run locally), and eventually the jobs timeout (4 hour job max time); I then get the zerolog: could not write event: ... error.

concat_tables.py.zip

[nick-youngblut]

I've asked this before on the Nextflow Slack workspace, but is there a better way of concatenating many files, while also linking those files to metadata? I created concat_tables.py because I could not find a better way.

To rephrase the question: nf-core pipelines generally use the pattern: tuple val(metadata), path(my_file), but when one wants to aggregate all files from the my_file channel in the tuple (e.g., via collect()), how does one map the per-file metadata to the target file in the aggregation process? I'm currently mapping the metadata by embedding it into the file names, but this has obvious limitations (e.g., special characters and selecting a "good" delimiter).

[jordeu]

Did you give it a try with process.scratch = false ? With Fusion v2 we are trying to solve the typical problem of needing to dimension the local FS of an instance a priori without knowing how much space will need a process. But to avoid this problem you need to run it with process.scratch = false.

If you didn't change anything, the default AWS AMIs only has a 30GB EBS slow boot volume for everything; SO, docker, process image, process input files, process output files, temporal files... even if it's a small pipeline it can get full very quickly. So on AWS Batch it is important to configure properly the queues, the defaults AMIs do not work for any real use case pipeline.

If you're evaluating or doing some tests, I suggest you to first use Tower Forge to create the AWS Batch compute environment for you because it's not an easy task to get things well configured even for small pipelines.

Example settings using Tower Forge with Fusion v2 enable:

[jordeu]

Check the .nextflow.log if there is more context to this error message. To have an idea where it comes from.

[nick-youngblut]

Below is the context in the nextflow log. There are 2 locations in the log with mentions of HTTP:

~> TaskHandler[id: 68; name: READ_QC_WF:BBMAP_DECON_MOUSE (10); status: RUNNING; exit: -; error: -; workDir: s3://pipeline-working-dir/strain_profiler/aws/work/6e/73f790742b83d2cbc62250ccc226f4]
Feb-21 23:23:08.637 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor awsbatch > tasks to be completed: 3 -- submitted tasks are shown below
~> TaskHandler[id: 62; name: READ_QC_WF:BBMAP_DECON_MOUSE (8); status: RUNNING; exit: -; error: -; workDir: s3://pipeline-working-dir/strain_profiler/aws/work/3c/3d9f9c69a9893ab2a347b63b2ffdca]
~> TaskHandler[id: 64; name: READ_QC_WF:BBMAP_DECON_MOUSE (9); status: RUNNING; exit: -; error: -; workDir: s3://pipeline-working-dir/strain_profiler/aws/work/b1/84cb673953aec8e50b125b944dd504]
~> TaskHandler[id: 68; name: READ_QC_WF:BBMAP_DECON_MOUSE (10); status: RUNNING; exit: -; error: -; workDir: s3://pipeline-working-dir/strain_profiler/aws/work/6e/73f790742b83d2cbc62250ccc226f4]
Feb-21 23:28:08.639 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor awsbatch > tasks to be completed: 3 -- submitted tasks are shown below
~> TaskHandler[id: 62; name: READ_QC_WF:BBMAP_DECON_MOUSE (8); status: RUNNING; exit: -; error: -; workDir: s3://pipeline-working-dir/strain_profiler/aws/work/3c/3d9f9c69a9893ab2a347b63b2ffdca]
~> TaskHandler[id: 64; name: READ_QC_WF:BBMAP_DECON_MOUSE (9); status: RUNNING; exit: -; error: -; workDir: s3://pipeline-working-dir/strain_profiler/aws/work/b1/84cb673953aec8e50b125b944dd504]
~> TaskHandler[id: 68; name: READ_QC_WF:BBMAP_DECON_MOUSE (10); status: RUNNING; exit: -; error: -; workDir: s3://pipeline-working-dir/strain_profiler/aws/work/6e/73f790742b83d2cbc62250ccc226f4]
Feb-21 23:32:32.359 [AWSBatch-executor-854] DEBUG nextflow.Session - Session aborted -- Cause: Unable to execute HTTP request: Read timed out
Feb-21 23:32:32.379 [AWSBatch-executor-854] DEBUG nextflow.Session - The following nodes are still active:
[process] READ_QC_WF:BBMAP_TRIM

Feb-21 19:54:57.857 [main] WARN  nextflow.trace.ReportObserver - Failed to render execution report -- see the log file for details
com.amazonaws.SdkClientException: Unable to execute HTTP request: pipeline-output.s3.us-west-2.amazonaws.com
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1216)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1162)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5453)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5400)
        at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)
        at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6528)
        at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)
        at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)
        at com.upplication.s3fs.AmazonS3Client.putObject(AmazonS3Client.java:219)
        at com.upplication.s3fs.S3FileSystemProvider.createDirectory(S3FileSystemProvider.java:550)
        at java.base/java.nio.file.Files.createDirectory(Files.java:700)
        at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
        at java.base/java.nio.file.Files.createDirectories(Files.java:753)
        at nextflow.trace.ReportObserver.renderHtml(ReportObserver.groovy:281)
        at nextflow.trace.ReportObserver.onFlowComplete(ReportObserver.groovy:140)
        at nextflow.Session.notifyFlowComplete(Session.groovy:1075)
        at nextflow.Session.shutdown0(Session.groovy:702)
        at nextflow.Session.destroy(Session.groovy:644)
        at nextflow.script.ScriptRunner.shutdown(ScriptRunner.groovy:240)
        at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:135)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:354)
        at nextflow.cli.Launcher.run(Launcher.groovy:487)
        at nextflow.cli.Launcher.main(Launcher.groovy:646)
Caused by: java.net.UnknownHostException: pipeline-output.s3.us-west-2.amazonaws.com
        at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
        at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1367)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1301)
        at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
        at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
        at jdk.internal.reflect.GeneratedMethodAccessor172.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
        at com.amazonaws.http.conn.$Proxy28.connect(Unknown Source)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1343)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)
        ... 28 common frames omitted
Feb-21 19:55:12.166 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done

[jordeu]

The error java.net.UnknownHostException: pipeline-output.s3.us-west-2.amazonaws.com seems to indicate some problem accessing pipeline-output S3 bucket.

[nick-youngblut]

I definitely have access to that s3 bucket, especially given that the pipeline writes many files to that bucket prior throwing the zerolog: could not write event error (that error occurs for processes near the end of the pipeline).
So, I don't think that s3 permissions issues are the problem.

[jordeu]

Ok, I don't known, but it really seems a networking issue. Maybe is only happening sporadically. Can you try to run Nextflow in an EC2 instance? If it's also failing there, then the problem should be something else.

[nick-youngblut]

Does Fusion2 & Wave require a special compute environment that is different from what is described at https://www.nextflow.io/docs/latest/aws.html?highlight=batch#aws-batch?

My pipelines seem to run correctly if I set:

  fusion.enabled  = false
  wave.enabled    = false

Maybe the issue is with how I set up my AMI or with the VPC settings associated with the Compute Environment?

[jordeu]

The permissions shouldn't be different. I'm always using a Tower Forge created environment, but you can check this section for more details on AWS policies required https://help.tower.nf/22.4/compute-envs/aws-batch/#manual

A part from the permissions issues, the default 30Gb EBS gp2 volume provided by AWS AMIs is not going to work with Fusion. The easy way, but still you'll get good performance (similar to an FSx), is to create an AMI that uses a 100GB EBS gp3 ~300MB/s volume. But if you want better performance then you need to go to format and mount NVMe disks.