Nextflow fusion with NFCore rnaseq running out of space

[Solyris83]

Hi,

I am not sure if this is a Nextflow issue or NFCORE issue, so please do tell if I am at the wrong place.

I am trying trying out the Nextflow fusion functionality on a nfcore rnaseq v3.9 run with everything coming from a single bucket

1. reference fasta/GTF : s3:///genome/grch38.fa + gencode.gtf
2. input fastq in sample sheet: s3:///test_data/sampleX_{1,2}.fastq.gz
3. work dir : s3:///work/

This is ran on an AWS EC2 machine.
nextflow run nf-core/rnaseq -r 3.9 -profile docker -resume --input /mnt/sdd/TEST_fusion/samplesheet.csv --outdir rnaseq_out_tagFUSION1 --gencode --fasta 's3://rnaseqtempbucket/project/misc_files/genome/GRCh38.primary_assembly.genome.fa' --gtf 's3://rnaseqtempbucket/project/misc_files/genome/gencode.v40.primary_assembly.annotation.gtf' --aligner star_salmon --pseudo_aligner salmon --save_align_intermeds --stringtie_ignore_gtf --skip_markduplicates --skip_dupradar --rseqc_modules bam_stat,inner_distance,junction_annotation,junction_saturation,read_distribution --max_memory 128.GB --max_cpus 16 --max_time 240.h -c /mnt/sdd/TEST_fusion/rnaseq_v3.9_50mil_reads.config -work-dir 's3://rnaseqtempbucket/project/work/'

Config file with parameters relevant to fusion enabling can be found below

docker {
  enabled = true
}

fusion {
  enabled = true
  exportAwsAccessKeys = true
}

wave {
    enabled = true
}

aws {
    profile = '<profile>' // using profile
}

process.scratch = false

The NFCORE process errored out at the qualimap stage with error found below

`
Workflow execution completed unsuccessfully!
The exit status of the task that caused the workflow execution to fail was: 255.

The full error message was:

Error executing process > 'NFCORE_RNASEQ:RNASEQ:QUALIMAP_RNASEQ (RMC375)'

Caused by:
Process NFCORE_RNASEQ:RNASEQ:QUALIMAP_RNASEQ (RMC375) terminated with an error exit status (255)

Command executed:

unset DISPLAY
mkdir tmp
export _JAVA_OPTIONS=-Djava.io.tmpdir=./tmp
qualimap
--java-mem-size=29G
rnaseq

-bam RMC375.sorted.bam
-gtf gencode.v40.primary_assembly.annotation.gtf
-p strand-specific-reverse
-pe
-outdir RMC375

cat <<-END_VERSIONS > versions.yml
"NFCORE_RNASEQ:RNASEQ:QUALIMAP_RNASEQ":
qualimap: $(echo $(qualimap 2>&1) | sed 's/^.QualiMap v.//; s/Built.$//')
END_VERSIONS

Command exit status:
255

Command output:
Initialized 800000 regions...
Initialized 900000 regions...
Initialized 1000000 regions...
Initialized 1100000 regions...
Initialized 1200000 regions...
Initialized 1300000 regions...
Initialized 1400000 regions...
Initialized 1500000 regions...
Initialized 1600000 regions...
Initialized 1700000 regions...
Initialized 1800000 regions...
Initialized 1900000 regions...
Initialized 2000000 regions...
Initialized 2100000 regions...
Initialized 2200000 regions...
Initialized 2300000 regions...
Initialized 2400000 regions...
Initialized 2500000 regions...
Initialized 2600000 regions...
Initialized 2700000 regions...
Initialized 2800000 regions...
Initialized 2900000 regions...
Initialized 3000000 regions...
Initialized 3100000 regions...
Initialized 3200000 regions...

Initialized 3284440 regions it total

Starting constructing transcripts for RNA-seq stats...
Finished constructing transcripts

Starting BAM file analysis

Sorting BAM file by name...

Read 10000000 records.
Read 20000000 records.
Read 30000000 records.
Read 40000000 records.
Read 50000000 records.
Read 60000000 records.
Read 70000000 records.
Read 80000000 records.
Read 90000000 records.
Read 100000000 records.
Read 110000000 records.
Read 120000000 records.
Read 130000000 records.
Finished reading inputs, merging and writing to output now.
Wed Aug 16 16:51:40 GMT 2023 WARNING Cleanup output dir

Command error:
Finished constructing transcripts

Starting BAM file analysis

Sorting BAM file by name...

Read 10000000 records.
Read 20000000 records.
Read 30000000 records.
Read 40000000 records.
Read 50000000 records.
Read 60000000 records.
Read 70000000 records.
Read 80000000 records.
Read 90000000 records.
Read 100000000 records.
Read 110000000 records.
Read 120000000 records.
Read 130000000 records.
Finished reading inputs, merging and writing to output now.
zerolog: could not write event: write /tmp/fusion.log4131595991: no space left on device
zerolog: could not write event: write /tmp/fusion.log4131595991: no space left on device
zerolog: could not write event: write /tmp/fusion.log4131595991: no space left on device
zerolog: could not write event: write /tmp/fusion.log4131595991: no space left on device
zerolog: could not write event: write /tmp/fusion.log4131595991: no space left on device
Failed to run rnaseq
net.sf.samtools.util.RuntimeEOFException: Premature EOF; BinaryCodec in readmode; streamed file (filename not available)
at net.sf.samtools.util.BinaryCodec.readBytes(BinaryCodec.java:373)
at net.sf.samtools.util.BinaryCodec.readBytes(BinaryCodec.java:357)
at net.sf.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:200)
at net.sf.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:37)
at net.sf.samtools.util.SortingCollection$FileRecordIterator.advance(SortingCollection.java:467)
at net.sf.samtools.util.SortingCollection$FileRecordIterator.next(SortingCollection.java:458)
at net.sf.samtools.util.PeekIterator.peek(PeekIterator.java:67)
at net.sf.samtools.util.SortingCollection$PeekFileRecordIteratorComparator.compare(SortingCollection.java:490)
at net.sf.samtools.util.SortingCollection$PeekFileRecordIteratorComparator.compare(SortingCollection.java:487)
at java.base/java.util.TreeMap.put(TreeMap.java:550)
at java.base/java.util.TreeSet.add(TreeSet.java:255)
at net.sf.samtools.util.SortingCollection$MergingIterator.next(SortingCollection.java:406)
at net.sf.samtools.SAMFileWriterImpl.close(SAMFileWriterImpl.java:190)
at org.bioinfo.ngs.qc.qualimap.process.ComputeCountsTask.sortSamByName(ComputeCountsTask.java:183)
at org.bioinfo.ngs.qc.qualimap.process.ComputeCountsTask.run(ComputeCountsTask.java:488)
at org.bioinfo.ngs.qc.qualimap.process.RNASeqQCAnalysis.run(RNASeqQCAnalysis.java:68)
at org.bioinfo.ngs.qc.qualimap.main.RnaSeqQcTool.execute(RnaSeqQcTool.java:221)
at org.bioinfo.ngs.qc.qualimap.main.NgsSmartTool.run(NgsSmartTool.java:190)
at org.bioinfo.ngs.qc.qualimap.main.NgsSmartMain.main(NgsSmartMain.java:113)

Wed Aug 16 16:51:40 GMT 2023 WARNING Cleanup output dir
�[90m4:52PM�[0m �[32mINF�[0m shutdown filesystem start
�[90m4:52PM�[0m �[32mINF�[0m shutdown filesystem done

Work dir:
s3://rnaseqtempbucket/project/work/5c/d8ac044e30e4d2c89661335fce73fa

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run
`
Any help on configuration which I have missed will be greatly appreciated!