No space left on device

[yunfeiguo]

Bug report

Expected behavior and actual behavior

Expected successful execution over large number of input files on a validated workflow. Actual behavior is that nextflow aborted due to Oct-08 06:42:07.342 [Actor Thread 3842] ERROR nextflow.extension.DataflowHelper - @unknown org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.io.IOException: No space left on device

Steps to reproduce the problem

We are trying to process over 3 million files, each process will handle 5000, each output file will be ~3-4MB, /tmp has 100GB of space, workdir has 10s of TB of space, execution is done over UGE cluster.

Program output

Oct-08 06:42:07.342 [Actor Thread 3842] ERROR nextflow.extension.DataflowHelper - @unknown
org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.io.IOException: No space left on device
        at org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:421)
        at org.iq80.leveldb.impl.DbImpl.writeInternal(DbImpl.java:683)
        at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:649)
        at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:642)
        at nextflow.file.SortFileCollector.add(SortFileCollector.groovy:211)
        at nextflow.file.SortFileCollector$add.call(Unknown Source)
        at nextflow.extension.CollectFileOp.processItem(CollectFileOp.groovy:179)
        at sun.reflect.GeneratedMethodAccessor228.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1268)
        at groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1048)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1142)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
        at groovy.lang.Closure.call(Closure.java:412)
        at groovy.lang.Closure.call(Closure.java:428)
        at groovy.lang.Closure$call.call(Unknown Source)
        at nextflow.extension.DataflowHelper$_subscribeImpl_closure2.doCall(DataflowHelper.groovy:289)
        at sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
        at groovy.lang.Closure.call(Closure.java:412)
        at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120)
        at groovyx.gpars.dataflow.operator.DataflowOperatorActor.onMessage(DataflowOperatorActor.java:108)
        at groovyx.gpars.actor.impl.SDAClosure$1.call(SDAClosure.java:43)
        at groovyx.gpars.actor.AbstractLoopingActor.runEnhancedWithoutRepliesOnMessages(AbstractLoopingActor.java:293)
        at groovyx.gpars.actor.AbstractLoopingActor.access$400(AbstractLoopingActor.java:30)
        at groovyx.gpars.actor.AbstractLoopingActor$1.handleMessage(AbstractLoopingActor.java:93)
        at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:132)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: No space left on device
        at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
        at sun.nio.ch.FileDispatcherImpl.writev(FileDispatcherImpl.java:72)
        at sun.nio.ch.IOUtil.write(IOUtil.java:148)
        at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:239)
        at java.nio.channels.FileChannel.write(FileChannel.java:414)
        at org.iq80.leveldb.table.TableBuilder.writeBlock(TableBuilder.java:199)
        at org.iq80.leveldb.table.TableBuilder.flush(TableBuilder.java:162)
        at org.iq80.leveldb.table.TableBuilder.add(TableBuilder.java:148)
        at org.iq80.leveldb.impl.DbImpl.buildTable(DbImpl.java:993)
        at org.iq80.leveldb.impl.DbImpl.writeLevel0Table(DbImpl.java:954)
        at org.iq80.leveldb.impl.DbImpl.compactMemTableInternal(DbImpl.java:918)
        at org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:465)
        at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:436)
        at org.iq80.leveldb.impl.DbImpl.access$100(DbImpl.java:85)
        at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:404)
        at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:398)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        ... 3 common frames omitted
Oct-08 06:42:07.379 [Actor Thread 3842] DEBUG nextflow.Session - Session aborted -- Cause: java.io.IOException: No space left on device
Oct-08 06:42:07.394 [Actor Thread 3842] DEBUG nextflow.Session - The following nodes are still active:
[process] calculate_frequency
  status=ACTIVE
  port 0: (queue) OPEN  ; channel: input_list
...

Environment

• Nextflow version: 21.04.3 build 5560
• Java version: 1.8.0_92-b14
• Operating system: Linux
• Bash version: 4.2.46(2)-release (x86_64-redhat-linux-gnu)

Additional context

(Add any other context about the problem here)