Looping over same data

[RenzoTale88]

Good morning,
I am currently working on a workflow using the DSL2 syntax.
I need to iteratively run an analysis an arbitrary number of times defined by the user.
The problem is, the output of iteration num. 1 would be used to as input of iteration num. 2, and so on until reaching the number defined by the user. Is there a straight-forward way to do it?
Just to give an idea, I'm thinking of something in the line of:

// Define inputs from a previous workflow
input1_ch = previous_proc.out[0]
input2_ch = previous_proc.out[1]
// Iterate same process over, replacing the input channels after every iteration
each x from 1..$params.iterations {
    process(input1_ch, input2_ch)
    input1_ch = process.out[0]
    input2_ch = process.out[1]
}

Thank you in advance for any help,
Andrea

[bobamess]

Similar to my response to Issue #2042, I would suggest using a bash wrapper script, e.g.

#!env bash
# bash script to repeatedly call a Nextflow workflow called repeat.nf 
# maxN times
#
# Usage:
# iter8NF.sh <maxN>
# e.g. to run maximum of 5 times
# iter8NF.sh 5
#
runN=1
maxN=$1
while [ "$runN" -le "$maxN" ]; do
nextflow run repeat.nf  --runN "$runN"  --maxN "$maxN" 
	runN=$((runN + 1))
done

In nextflow.config, include default values (maxN will get overwritten by whatever is used in the command line) with

params {

	runN = 1
	maxN = 2
}

In repeat.nf:

• include a test to catch when runN == 1 so that the input channel(s) can be initiated instead of using output from previous run. Alternatively, have a separate workflow for the first run and add an if [ test ]; then … else … fi block to the wrapper script to control which workflow is called.
• save the output(s) either by collecting into a .txt/.csv file(s), or saving to a relevant file format.
• passing the value of maxN may be useful if there are any outputs that need to be saved for each run that would otherwise get overwritten.
  If a condition is met, such that not all maxN runs need to be executed, then a test for this condition can be added to the while [ test ]; do ... done loop, e.g. testing for existence of a report file produced as optional output from the Nextflow workflow when this condition is reached, or similar, and adding a break command.

It seems Nextflow isn’t designed with iteration in mind, however, I've found that using a bash wrapper script can easily overcome this limitation, while retaining all of the advantages of parallelization etc. that Nextflow's DSL2 has to offer.

[RenzoTale88]

@bobamess thank you for your suggestion, it looks like a reasonable workaround to the problem. Nevertheless, I'd still like to see the function implemented, since at the moment it comes with some limitations (e.g. if I'm running on a distributed system it becomes tricky to configure mixed local and remote execution). Plus, you have to manage in effect two separate workflows, that comes as less convenient.