Workflow output definitions and schema

[bentsherman]

Spun off from #4042 (comment)

Currently, the publishDir directive is used to define "workflow outputs". It is a "push" model in which individual processes are responsible for deciding which of their outputs should be workflow outputs, and how to publish them. It was created in the days of DSL1 when there was no concept of workflows / subworkflows.

With DSL2 and the introduction of workflows, publishDir is an unwieldy way to define workflow outputs. For one thing, it is extremely difficult to get a full picture of a workflow's outputs because they are scattered across the process definitions. Additionally, a process could be invoked many different times by different workflows, and each workflow might have a different idea of its own outputs. All in all, it no longer makes sense to define workflow outputs in the process definition.

Instead, it should be possible to specify an output schema for a workflow, which defines the full set of published outputs, as well as metadata, in terms of the processes (and subworkflows) that the workflow invokes.

The two options discussed so far are:

1. define the workflow outputs as part of the workflow definition (e.g. an output block) which can reference the outputs of subcomponents
2. define the workflow outputs in a separate file similar to what is done for nf-core modules

See also #4661 for discussion of where to put additional publish options such as mode and saveAs. The publish mode could become a config option or process directive (e.g. publishOptions.mode) to allow for process-specific configuration, or they could become part of the output schema.

[pditommaso]

I see like this:

1. Adopt the meta.yml to define type of each process input & output. Currently, basic types are used but it could be extended to support structured used-defined types. Nextflow can read all of these and it can create a runtime view of the types and validate them before running the workflow.
2. Add a (top) level DSL to define the workflow outputs as a replacement for publishDir. This should allow to define and keep in a single place the workflow output definition and configuration (more below_
3. Add an (*optional*) output schema having a similar structure to the current input schema. When existing the output definition (2) much match the schema definition.

Modelling workflow outputs

The point at 2 is the core. The goal is to provide a way to define the structure of the output that be produced by a pipeline execution in a concise and expressive way, and keep this into a single place.

A core pattern that needs to be addressed is how to define a semantic-directory structure collection of some or all workflow output files and give the ability to create (optionally) one or more index files; that's another very common practice.

For the sake of this example, let's borrow the recent introduced topic channel. This allows collecting all output files created during the pipeline execution having the same semantic (or just topic name).

We could use this concept (syntax purely hypothetical) to write something like this

workflow {
  rnaseq(params.something) 
} 

output {
  directory = params.outdir 

   samples {
      topic 'someName'
      path 'subdir/where/store/the/files'
      pattern '*.assembly.fa.gz' 
      index {
         type 'csv'
         path 'where/to/store/samples.csv'
      }
   }

   moreHere {
     ..
   }
}

The type information defined at 1 (i.e. meta.yml) and associated with the specified topic could be used to define and render the structure of the index file and carry out other type validation.

The overall output definition could be validated against the output JSON schema if provided

[bentsherman]

I like the idea of using the channel topics to hold workflow outputs.

One thing that's missing from your example is the metadata. Once we have record types this will be easier, but for now I think we'll need some way to translate the standard tuple-with-meta-map pattern into a record for the index file, so that the metadata can be saved as columns.

I think we can add some config to this output DSL to control this mapping, with some sensible default for the most common use case.

[pditommaso]

One thing that's missing from your example is the metadata

I believe this needs to be inferred from the meta.yml (when provided), but yes, it could be done in a separate effort.

[bentsherman]

I feel like the meta.yml inputs and outputs are mostly duplicating the process definition though. I would want the type information to be defined in the process

[marcodelapierre]

From the chat Paolo and I had on Monday, I would say one key aspect of point 1 on meta.yml is the module-level character of some key information: list of inputs/outputs, their types, corresponding file patterns, eventually other related metadata.

This work package to upgrade the definition of outputs at the workflow-level should leave that info at module level, and ensure it is defined in such a way that can then be processed to match the workflow level output definition (point 2).

I agree, Ben, that ideally the output types should be in the process def. For point 2. above, probably everything could then be sourced from the process def, I wonder.

--

However, there is also point 3 on the output schema. Again, to preserve modularity, key process-level information should be defined at module-level, gathered and processed together with knowledge from 2. to produce the workflow output schema. Where would this module-level info go? The process definition would probably become too cluttered, and that's where the meta.yml is handy.

And here comes a second key point that emerged during the Monday chat: is JSON a good format for these schemas? do we have room to consider alternatives? (nf-core requirements?) YAML could be a more readable and approachable alternative to pipeline devs.

--

Question to close, going back to Ben's comments: shall we prioritise work to support Types in the process definition as a prerequisite to this one? Or can we start working on the re-design of the workflow-level output definition (point 2.), and defer Type support to later?

[pditommaso]

I feel like the meta.yml inputs and outputs are mostly duplicating the process definition though. I would want the type information to be defined in the process

I agree with that, but having a static type system in the language will require some time. My proposal is to rely on types the meta.yml (and other metadata) until we don't a proper type system in the language, and make sure it will be possible to transition from a file base metadata to a fully static lang check in the future.

However, there is also point 3 on the output schema. Again, to preserve modularity, key process-level information should be defined at module-level, gathered and processed together with knowledge from 2. to produce the workflow output schema. Where would this module-level info go?

I believe the approach proposed allows us to combine modularity with overall output and schema definition.

shall we prioritise work to support Types in the process definition as a prerequisite to this one?

Think I have answered above

[bentsherman]

Honestly I would rather not deal with types at all. I thought we agreed it was a separate effort from workflow outputs.

Also, if we use channel topics to facilitate the workflow outputs then we don't really need the meta.yml, because the process definition will specify the target topic for each output.

The 99% use case for metadata is as a tuple with a meta map as the first element. When an output is structured in this way, it is trivial to transform it into a record for the index file.

[pditommaso]

Honestly I would rather not deal with types at all. I thought we agreed it was a separate effort from workflow outputs.

Agree, I'll open a separate ticket to track it independently

[bentsherman]

Okay I think I have enough clarity now to draft a prototype for the outputs DSL proposed by Paolo. I will aim to have it ready by the end of next week.

[marcodelapierre]

A comment on Paolo's concept above.

Do we need to use the topic? or could we instead name a list of those processes, that are part of the workflow and that create the files that may match the pattern for that specific output category?
It would make the overall syntax more concise, by reusing process names instead of having to add topic syntax across the processes.

[bentsherman]

Well, I really like the topic approach because it's intuitive and should be easy for users to transition to it from publishDir. Of course, workflow outputs are still defined in part at the process level, but not nearly as much as before.

With this outputs DSL, I could search the topic name over the project directory and get a view of the processes that push to that topic, whereas currently I have no such top-level identifier.

One issue with listing the processes is that we need to target specific outputs, not just the process. We could technically do this with the emit names, but this is much more verbose and I'm willing to bet that many users would lean towards using the topics.

Let's try the topic approach (should be easy to implement) and see how it looks with rnaseq