Idiomatic way of gathering stats throughout a workflow?

[ameintjes]

I'm looking for a clean way to collect stats about a dataset as it progresses through a workflow.

In our example we're passing VCF data through numerous processes, and tweaking filters, order of execution etc.

It would be extremely helpful to be able to insert bcftools stats at different points and send the output downstream for collation into a report at the end... ideally in a way that that does not clutter the workflow code itself too much.

It seems to me the ideal pattern/implementation:

• would be easy to toggle on/off with a config parameter
• should require minimal code to insert a new "checkpoint" (i.e. no need to adjust the report creation logic downstream)
• would be in a process because it may be compute heavy

The best to solution we've come up so far is a process that takes VCF as input, runs bcftools stats on it then stores the results using the publishDir directive. This works well, except that we have to import new aliases for every time we invoke it.

Has anyone else found a more elegant solution?

[mribeirodantas]

Hello, @ameintjes. Can't you run it once at the end with the output of all the processes that you're interested in?

[ameintjes]

That hadn't occurred to me... will give it a try

thanks!

[mribeirodantas]

Cool!