Allow for dynamic resolution of the storeDir for collectFile

[stevekm]

New Feature Request

The collectFile Channel operator has the ability to dynamically set the file name argument from within a closure like this;

map_ch = Channel.of(
        ["id":"S1", "val":10],
        ["id":"S1", "val":20],
        ["id":"S2", "val":30],
        ["id":"S2", "val":40])

map_ch.collectFile(storeDir: "output", newLine: true) { meta ->
        def filename = "${meta.id}.txt"
        return [filename, meta.val.toString()]
    }

// output/S1.txt
// output/S2.txt

However it would be really helpful to be able to save files in this manner with dynamically set sub-directories as well. Instead of saving to

output/S1.txt
output/S2.txt

I want to be able to save to

output/S1/S1.txt
output/S2/S2.txt

It does not seem like its currently possible to do this. You cannot use any dynamic methods in the storeDir arg, and trying to put a subdir in the name arg seems to break often, especially with the -resume feature;

map_ch.collectFile(storeDir: "${params.outdir}") { meta ->
        def filename = "${meta.id}/${meta.id}.txt"
        return [filename, meta.val.toString()]
    }

ERROR ~ No such file or directory: output/S1/S1.txt

Is there any way we could get a feature like this? It would help a lot with collection of files and metadata from process outputs that we want to keep associated with e.g. per-sample output subdirectories. The only other way I have been able to figure out so far for outputting files in per-sample subdirs is to use a Nextflow process but that is not ideal if we can just do it with collectFile instead

[mribeirodantas]

You could create the folder from within your closure. I added new File("output/${meta.id}").mkdirs() to your code below and it works.

map_ch = Channel.of(
        ["id":"S1", "val":10],
        ["id":"S1", "val":20],
        ["id":"S2", "val":30],
        ["id":"S2", "val":40])

map_ch.collectFile(storeDir: "output", newLine: true) { meta ->
        new File("output/${meta.id}").mkdirs()
        def filename = "${meta.id}/${meta.id}.txt"
        return [filename, meta.val.toString()]
    }

Folder structure:

tree output/
output/
├── S1
│   └── S1.txt
└── S2
    └── S2.txt

3 directories, 2 files