Processes for imported workflows receive incorrect inputs

[hotplot]

Bug report

Expected behavior and actual behavior

I have created a workflow in a module file that runs three processes, one after the other. Each process is passed the same input that was passed to the workflow, does some processing, and saves some output. The final process in the workflow takes the same input, plus the outputs of the preceeding processes, and saves some additional output.

I then import that workflow into main.nf and apply it to a channel with multiple values (so that the workflow is run multiple times, once per input). Call the inputs a, b, and c.

If the workflow is called with a as input, and passes that same a to the first two processes, I would expect the third process to be called with a and the output of the two processes that were given a as inputs. Instead, it gets called with a and the output of a random combination of processes applied to a, b or c.

That means the final process is being applied to an invalid combination of inputs. Some of the inputs are for completely unrelated runs/instances of the workflow.

Steps to reproduce the problem

Repo with runnable test case: https://github.com/hotplot/nf-wf-issues

Program output

In the workdir for each of the final processes run by the workflow, there should be:

1. The input file passed to the entire workflow (i.e. a)
2. A directory output_p1 containing the same file
3. A directory output_p2 containing the same file

Instead, the two output subdirectories contain a, b, or c at random (and rarely the same file as is found in the parent dir)

Environment

• Nextflow version: 20.10.0
• Java version: openjdk version "1.8.0_302", OpenJDK Runtime Environment (build 1.8.0_302-b08), OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode)
• Operating system: Linux
• Bash version: GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)

Additional context

[pditommaso]

That's expected. As per the parallel execution of p1 and p2 the sequence of the output values/files cannot be expected to be the same as they were received.

https://github.com/hotplot/nf-wf-issues/blob/master/foo.nf#L55

Therefore the assumption in the above line that p3 is receiving [a,b,c] as input and from corresponding p1.out and p2.out outputs is not correct.

[hotplot]

Thanks for your response, and for clarifying that this is at least by design rather than a bug.

The type of problem I'm trying to solve is basically:

1. A NF workflow takes some data as input and runs some processes in parallel and some sequentially on that data, and eventually returns some final output
2. The workflow is run multiple times, in parallel, on different and distinct inputs
3. The final outputs of each instance of the workflow are collected and used as an combined input to a downstream process

Conceptually, it seems logical that there should be no mixing of inputs or outputs between separate invocations of the workflow. If I start one instance of the workflow with an input of inputs/a and a second instance with an input of inputs/b then it doesn't make any sense to me that the second invocation should randomly see outputs from processes that were started by the first invocation.

I do not know how hard it would be to make this change, but I think it would be strictly an improvement over the current behavior. Users who currently rely on a workflow like this (if there are any) must be able to tolerate random recombination of results, so if it was changed to always be the "correct" combination that shouldn't break anything for them. But it would also allow people to use workflows this way even when they require the outputs to be combined correctly.

At the moment I have worked around the issue by using a single process (in place of the workflow) that runs each of the "workflow processes" sequentially in a bash script. That way each of the intermediate outputs is guaranteed to be from the correct instance of the process. However it doesn't make good use of the HPC resources, and if the cache is invalidated for any reason it results in a significant amount of recomputation. Is there a better approach I should consider?

[bentsherman]

Hi @hotplot , processes don't necessarily produce outputs in the same order as inputs, so if you want to combine the outputs of multiple processes (as you do in process c), you need to add some metadata to each channel (e.g. a sample id) so that you can recombine the outputs properly based on a matching key. Simply calling process c with multiple channels like that is equivalent to using the merge operator, which is discouraged for the same reason.