How To Specify a Different publishDir for Each Execution of a Process

[jeffquinn-msk]

This question is best asked via illustrative example. Assume this situation where we have process CREATE_FILE which creates an output file, a subworkflow that takes a single input, a number which is written to the file in CREATE_FILE, and a top level workflow which calls SUBWORKFLOW with a channel of numbers as input:

process CREATE_FILE {
  publishDir "${params.outdir}"

  input:
    val number

  output:
    path the_file

  """
  echo "${number}" >> the_file
  """
}

workflow SUBWORKFLOW {
  take:
    number

  main:
    CREATE_FILE( number )

}

workflow {
  var numbers = Channel.from([1,2,3])
  SUBWORKFLOW( numbers )
}

The problem with this is:

$ nextflow run example.nf --outdir /tmp/example
N E X T F L O W  ~  version 23.04.1
Launching `example.nf` [evil_kalman] DSL2 - revision: cec2b48335
executor >  local (3)
[6f/065d54] process > SUBWORKFLOW:CREATE_FILE (3) [100%] 3 of 3 ✔

$ cat /tmp/example/the_file
3

publishDir is constant with respect to the iteration through the channel, so each subsequent iteration clobbers the output from the previous iteration. The obvious solution would be to make the publishDir dynamic with respect to the iteration, but I cannot for the life of me figure out what the best mechanism is to do this. Please advise.

Thanks,

Jeff

[bentsherman]

You can either make the output file name different in each task execution (e.g. based on the task index or provide a custom id), or use the saveAs option in publishDir to do the same.

https://nextflow.io/docs/edge/process.html#publishdir

[jeffquinn-msk]

Thanks for the quick response.

So something like this would be a reasonable approach?

process CREATE_FILE {
  publishDir = "${params.outdir}"
  input:
    val number

  output:
    path "${number}/the_file"

  script:
    """
    mkdir -p "${number}"
    echo "${number}" >> "${number}/the_file"
    """
}

workflow SUBWORKFLOW {
  take:
    number

  main:
    CREATE_FILE( number )

}


workflow {
  var numbers = Channel.from([1,2,3])
  SUBWORKFLOW( numbers )
}

Resulting in:

$ nextflow run example.nf --outdir /tmp/example
N E X T F L O W  ~  version 23.04.1
Launching `example.nf` [peaceful_koch] DSL2 - revision: 3b7b4eb1be
executor >  local (3)
[c5/965701] process > SUBWORKFLOW:CREATE_FILE (3) [100%] 3 of 3 ✔

$ ls -R /tmp/example
1	2	3

/tmp/example/1:
the_file

/tmp/example/2:
the_file

/tmp/example/3:
the_file

[bentsherman]

Yes exactly