Using combine with remainder

[glormph]

Hi,
I'm trying to get a join on two channels with non-unique keys, which by analogy with SQL I would say is a LEFT or RIGHT OUTER JOIN, so:

left  = Channel.of(['X', 1], ['Y', 2])
right = Channel.of(['Z', 6], ['Y', 5], ['X', 4], ['X', 7])

And I'd like the output to be:

[X, 1, 4]
[Z, null, 6]
[Y, 2, 5]
[X, 1, 4]

I can't really figure out how to do this, since .join(..., remainder: true) seems to only match one record for X, the second one coming out with null, and .combine(..., by: 0) does match multiple but it does not have remainder, so the Z record is lost.

Am I doing this wrong? It sounds like a usecase too basic to not already somehow be possible.

[bentsherman]

You're right, combine doesn't have the option to emit remainders at the end like join, although it could in theory. I guess no one else has requested it until now.

I'd be curious to know more about what you need it for? I don't think I've ever had to compute an outer product and keep the remainder, but I'm always open to learning new things ...

[glormph]

Bear with me, since I usually develop proteomics things, but have an inherited genomics pipeline to work on and I don't fully understand everything in it. I have some VCF files (multiple per sample id) that have to be annotated with transcripts from GTF files (one per sample id). The catch is that sometimes a user doesn't have transcript info and it'd be nice if that can differ per sample.

vcfs = Channel([s1, s1_indel.vcf], [s1, s1_snv.vcf], [s2, s2_indel.vcf], [s3, ...], ...)
gtfs = Channel([s1, s1.gtf], [s2, s2.gtf])

My intuitive way to go about this would be:

vcfs
| combine(gtfs, by: 0, remainder: true)
| branch {
  withtranscript: it[-1] != null
  notranscript: it[-1] == null
}
| set { vcfs_gtfs }

vcfs_gtfs.withtranscript
| annotateVcfs
| set { annotated }

vcfs_gtfs.notranscript
| map { [ it[0], it[1] ] }
| concat(annotated)
| Do_Stuff

I'm not sure if I'm thinking too convolutedly about this, and I've worked around it somehow, but this was what I was after.

[bentsherman]

That makes sense, you're just doing a join between the VCF and GTF files.

Well, you could group the VCF files by sample id and turn the combine into a join:

vcfs
| groupTuple
| join(gtfs, remainder: true)
| transpose
| branch {
  withtranscript: it[2] != null
  notranscript: it[2] == null
}
| set { vcfs_gtfs }

The transpose splits them up again, since I assume that you want to analyze each VCF file individually.

But the drawback here is that, if you don't know the expected number of VCF files per sample, the groupTuple won't emit any outputs until it has processed all of its inputs, which can be a major bottleneck at scale. If you do know the size then I can show you how to encode it with groupKey(). On the other hand, combine() can simply emit each new combination immediately, so maybe there's a case for it.

Let me know what you think, like do you need to analyze each VCF file individually? Do you know the expected number of VCFs per sample?

[glormph]

Neat! Hadn't thought about groupTuple here actually. I do know the size, when starting the pipeline the samples/VCFs are defined in the input. So I guess I can parse the input file up front and create a sample/size map for it that groupKey can address. Thanks!