Output files (chromosomal chunks) merging in nextflow

[zillurbmb51]

I have a nextflow process that produces multiple chunks for each chromosomes into a channel, say, imputation which looks like,

chr1.imputed.chunk1.gen.gz chr1.imputed.chunk2.gen.gz chr1.imputed.chunk3.gen.gz 
chr1.imputed.chunk1.stats chr1.imputed.chunk2.stats chr1.imputed.chunk3.stats
chr1.imputed.chunk1.bgen chr1.imputed.chunk2.bgen chr1.imputed.chunk3.bgen
.....
chr2.imputed.chunk1.gen.gz chr2.imputed.chunk2.gen.gz chr2.imputed.chunk3.gen.gz 
chr2.imputed.chunk1.stats chr2.imputed.chunk2.stats chr2.imputed.chunk3.stats
chr2.imputed.chunk1.bgen chr1.imputed.chunk2.bgen chr2.imputed.chunk3.bgen
......

There are many chunks for each chromosome (22 chromosomes). How can I effectively merge them to respective chromosomes for each type of file set to get,

chr1.imputed.merged.gen.gz
chr1.imputed.merged.stats
chr1.imputed.merged.bgen
......
chr2.imputed.merged.gen.gz
chr2.imputed.merged.stats
chr2.imputed.merged.bgen

Any help?

[colindaven]

Can't you use the collectFile operator, documented in nextflow patterns ?

https://nextflow-io.github.io/patterns/collect-into-file/

[mribeirodantas]

With the two command below...

touch chr1.imputed.chunk1.gen.gz chr1.imputed.chunk2.gen.gz chr1.imputed.chunk3.gen.gz chr1.imputed.chunk1.stats chr1.imputed.chunk2.stats chr1.imputed.chunk3.stats chr1.imputed.chunk1.bgen chr1.imputed.chunk2.bgen chr1.imputed.chunk3.bgen chr2.imputed.chunk1.gen.gz chr2.imputed.chunk2.gen.gz chr2.imputed.chunk3.gen.gz chr2.imputed.chunk1.stats chr2.imputed.chunk2.stats chr2.imputed.chunk3.stats chr2.imputed.chunk1.bgen chr2.imputed.chunk2.bgen chr2.imputed.chunk3.bgen

... I have created a folder that contains the following files (I'm working with files to make sure it's easy for you to plug my solution into your pipeline):

➜  sandbox tree .
.
├── chr1.imputed.chunk1.bgen
├── chr1.imputed.chunk1.gen.gz
├── chr1.imputed.chunk1.stats
├── chr1.imputed.chunk2.bgen
├── chr1.imputed.chunk2.gen.gz
├── chr1.imputed.chunk2.stats
├── chr1.imputed.chunk3.bgen
├── chr1.imputed.chunk3.gen.gz
├── chr1.imputed.chunk3.stats
├── chr2.imputed.chunk1.bgen
├── chr2.imputed.chunk1.gen.gz
├── chr2.imputed.chunk1.stats
├── chr2.imputed.chunk2.bgen
├── chr2.imputed.chunk2.gen.gz
├── chr2.imputed.chunk2.stats
├── chr2.imputed.chunk3.bgen
├── chr2.imputed.chunk3.gen.gz
├── chr2.imputed.chunk3.stats
└── main.nf

1 directory, 19 files

I will now run a loop in bash to write to every file, its own filename. This way, by the end, we should be able to verify that the right files have been merged. Do this with the code below:

for f in *chunk*
do
  echo $f > $f
done

I will read these files into a channel and use the map channel operator so that every element of the channel is a tuple with ID (chromosome number), extension type and filename. With that done, I will use the branch channel operator to split my channel into three channels, one for each filetype. Then, I will use collectFile to merge the content of each file into a merged version of it that I will name using the chromosome number + extension type, e.g. chr1_stats.txt. See the code below:

workflow {
  Channel
    .fromPath("*imputed*", checkIfExists: true)
    .map { it -> [it.name.split('.imputed')[0], it.name.split('chunk*..')[1], it]}
    .branch {
      stats: it[1] == 'stats'
      bgen: it[1] == 'bgen'
      gengz: it[1] == 'gen.gz'
    }
    .set { cfiles_ch }
  // We have three channels now, I will do for stats and you can repeat it
  // yourself for the other two channels
  cfiles_ch
    .stats
    .map { id, extension, file -> [id, file] }
    .collectFile() { id, file ->
      [ "${id}_stats.txt", file.text]
    }
}

There won't be any interesting output, as what matters to us was written to a file. You can check your work directory to find what was done with tree:

➜  sandbox tree work
work
└── tmp
    └── db
        └── 7ed3d7687dc0ad87ce76cc9290d3fe
            ├── chr1_stats.txt
            └── chr2_stats.txt

4 directories, 2 files

Let's cat chr1_stats.txt, for example:

➜  sandbox cat work/tmp/db/7ed3d7687dc0ad87ce76cc9290d3fe/chr1_stats.txt
chr1.imputed.chunk2.stats
chr1.imputed.chunk1.stats
chr1.imputed.chunk3.stats

And voilà :).

A reminder that you can use the publishDir process directive in your process if you want these files to be stored somewhere.