Replacement for merge?

[multimeric]

It seems that merge is now deprecated. However there doesn't seem to be a simple replacement, as join requires a key. If I want to join two channels by position, how can I now do that? ie how can I replicate this example from the documentation?

odds  = Channel.from([1, 3, 5, 7, 9])
evens = Channel.from([2, 4, 6])

odds
    .merge( evens )
    .view()

[1, 2]
[3, 4]
[5, 6]

[pditommaso]

This was done on purpose because merge when used along with processes was source of non-deterministic executions

[multimeric]

I see. Unfortunately it isn't trivial to simply avoid merging by position. For example, if we have a process with two output channels, how group together the outputs from each execution? Especially if each channel has primitive values in it so there is no way to join using any other method than position:

nextflow.enable.dsl=2

process test {
    input:
        val x
    output:
        val x2
        val x3
    exec:
        x2 = x ** 2
        x3 = x ** 3
        """
        true
        """
}
workflow {
    result = test(Channel.from(1, 2))
    result[0].view()
    result[1].view()
    // Now, make a channel that emits [1, 1], [4, 8] 
}

[multimeric]

Further, this doesn't even solve the non-determinism. Just running a process with two input channels implicitly "merges" them together, so it's trivial to create a non-deterministic pipeline without merge. For example:
multiply.nf:

nextflow.enable.dsl=2
process multiply {
    input:
        val a
        val b
    output:
        val c
    exec:
        c = a * b
}

ordering.nf:

nextflow.enable.dsl=2
include {multiply; multiply as multiply_2} from './multiply'
workflow {
    a = Channel.from(1, 2, 3, 4)
    b = Channel.from(1, 2, 3, 4)
    c = multiply(a, b)
    d = multiply_2(c, a).view()
}

Outputs (at different times):

4
64
3
18

16
18
3
16

1
18
12
64

[bcantarel]

I agree join does not replace merge. I have 2 channels -- one with fastq files downloaded by SRR using their project ID and another with the run ids for those fastq files and I would like to use merge to create one channel with the id and the fastq files...

process getfastq_data {
label 'basic'
publishDir "$params.output/${sid}/fastq", mode: 'copy'
input:
val id
output:
path ("design.txt"), emit: design
path("*.fastq.gz"), emit: fq
path("runids.txt"), emit: runid
when:
id
script:
"""
bash ${repoDir}/scripts/get_sequence_data/get_fastq_sra.sh -i ${id}
ls *.fastq.gz | cut -f 1 -d '_' > runids.txt
"""
}
runid
.splitText()
.map{it -> it.trim()}
.merge(fq)

[multimeric]

My workaround for that particular situation is to output a tuple for the channel you want to group together. You can always then split the tuple but you can't do the opposite now that merge is dead:

output:
    tuple path("design.txt"), path("*.fastq.gz"), path("runids.txt")

[bcantarel]

I am not sure I understand the channel I need has runID, SampleID and Files:
[SRR1929649M10, SAMN03447157, [/path/SRR1929649M10_1.fastq.gz, /path/SRR1929649M10_2.fastq.gz]]
[SRR1929796M10, SAMN03447157, [/path/SRR1929796M10_1.fastq.gz, path/SRR1929796M10_2.fastq.gz]]
[SRR1929662M10, SAMN03447157, [/path/SRR1929662M10_1.fastq.gz, /path/SRR1929662M10_2.fastq.gz]]
[ERR2260495M30, SAMEA104469176, [/path/ERR2260495M30_1.fastq.gz, /path/ERR2260495M30_2.fastq.gz]]

The one you suggest is a file with the run id and sample id mapping (2 columns tab delimited file), one with the sample id (1 id on each line) and the list of files, how do I use split to create the channel I need for the analysis steps?

[multimeric]

I don't fully understand your question, but if one of the fields you need is inside a file you can use splitCsv(elem: i) to split the ith item of a tuple into multiple rows.

So if the channel outputs:

[runid1.tsv, [reads_11.fq, reads_22.fq]]
[runid2.tsv, [reads_21.fq, reads_22.fq]]

Then you splitCsv(elem: 0):

[["run_id_1", "sample_id_1"], [reads_11.fq, reads_22.fq]]
[["run_id_2", "sample_id_2"], [reads_21.fq, reads_22.fq]]

Then you can just map it back to a more reasonable structure.

[pditommaso]

Since the concerns raised in this thread and other conversions, it has been decided to not deprecate the merge operator anymore and put a clear notice in the documentation.

1649944

[Kibubu]

nextflow/modules/nextflow/src/main/groovy/nextflow/extension/OperatorImpl.groovy

Lines 1335 to 1347 in aed6c23

	@DeprecatedDsl2
	DataflowWriteChannel merge(final DataflowReadChannel source, final DataflowReadChannel other, final Closure closure=null) {
	final result = CH.createBy(source)
	final inputs = [source, other]
	final action = closure ? new ChainWithClosure<>(closure) : new DefaultMergeClosure(inputs.size())
	final listener = stopErrorListener(source,result)
	final params = createOpParams(inputs, result, listener)
	newOperator(params, action)
	return result;
	}
	// NO DAG
	DataflowWriteChannel merge(final DataflowReadChannel source, final DataflowReadChannel... others) {

It looks like merge is still deprecated in DSL2.

[bentsherman]

I know this discussion is old, but I was thinking about the merge operator recently and I'm wondering if anyone has an actual use case for it. When would you need to combine two channels where you don't care how they get paired up?

If you want to generate all combinations of two channels (e.g. outer product), use combine or cross.

If you want to join items from two channels with a matching key (e.g. inner product), use 'join.

If you want to emit multiple values from a process, but want those values to stay connected, then emit a tuple in a single channel.

If anyone does have a use case, it would be good to include in our patterns docs. Otherwise it would be best to discourage people from using it, citing the alternatives listed above.

[bentsherman]

Actually this is exactly what we're talking about in the other discussion. You are relying on the fact that multiMap preserves order. It may happen to do that, but it is not a requirement. The implementation of multiMap could change one day and it would break your code.

In this case, your process should take some_channel directly as input (although you'll have to convert to a tuple for now). You should never separate the metadata (id) from the data (fasta) and expect to be able to re-combine them correctly later on.

[multimeric]

Fair enough, but going off memory, if you make the process accept the whole channel as a tuple, it won't localise the files within the tuple. Or if it does do that, this behaviour isn't documented.

[bentsherman]

It does. Files within the tuple are treated just like standalone files. I agree that should be clearly documented. I had to figure it out by trial and error.

[multimeric]

But if it's in any other kind of structure like a map (using the val directive) it won't get localised, right?

[bentsherman]

That's correct, val inputs don't stage files. You could add a map operator before and after the process to convert between maps and tuples, if you wanted to use maps throughout your workflow.

I see you authored the issue about map outputs #2127. I'm going to pin that issue so it doesn't get lost. I think it would be a nice feature to have but need to see how feasible it would be.