Is there a away to prioritize final results?

[harohodg]

If I have a pipeline with three steps which are sequentially dependent (ie the output of step 1 is the input of step 2 etc) is there a way to ensure that steps 2 and 3 run as soon as their inputs are ready?

As a small test case

#!/usr/bin/env nextflow

nextflow.enable.dsl=2

process step1 {
    input:
        val num
    output:
        stdout
    script:
    """
    sleep \$((RANDOM % 2 + 1))
    echo num
    """
}

process step2 {
    input:
        val num
    output:
        stdout
    script:
    """
    sleep \$((RANDOM % 2 + 1))
    echo num
    """
}

process step3 {
    input:
        val num
    output:
        stdout
    script:
    """
    sleep \$((RANDOM % 2 + 1))
    echo $num
    """
}


workflow {
    numbers = channel.from(1..100)
    step3( step2( step1(numbers) ) ) | view
}

when runs produces something like

[19/baacf2] process > step1 (68)  [ 17%] 17 of 100
[-        ] process > step2       [  0%] 0 of 17
[-        ] process > step3       -

What I would like to see is something like

[19/baacf2] process > step1 (2)  [ 2%] 2 of 100
[23/b34dcf2] process > step2 (3)      [  50%] 1 of 2
[42/d4a5f6] process > step3    (4)   [  100%] 1 of 1

[pditommaso]

Interesting puzzle, not sure why this happens.

[harohodg]

How does nextflow decide which process to run next when using the local executor? The output looks like it's running in FIFO order.

[bentsherman]

The order is arbitrary. In theory it could execute them in depth-first or breadth-first order, or somewhere in between. If you are getting this behavior consistently, it's probably just an artifact of how the scheduling is implemented

nextflow/modules/nextflow/src/main/groovy/nextflow/processor/TaskPollingMonitor.groovy

Lines 596 to 600 in da5f759

	protected int submitPendingTasks() {
	int count = 0
	def itr = pendingQueue.iterator()
	while( itr.hasNext() && session.canSubmitTasks() ) {

The executor's pending queue is processed as FIFO, but the three processes can submit tasks to the pending queue concurrently, so I would expect a different ordering across runs.

[harohodg]

Ah, so if the input for step one is ready immediately and step one takes awhile to run then most of step one will be submitted to the pending queue ahead of any following steps.

With my own pipeline I was able to prioritize final results by using the clusterOptions directive with an older version of hyper queue "--priority #" and with slurm "--nice #" but it would be nice if it was natively built into nextflow. Perhaps as a "priority" directive.