Workflow state dependency

[ericart89]

Hello,

I have a workflow that consists of two different "subworkflows". The first workflow finishes with one process whose errorStrategy is to "ignore" because sometimes there are jobs that fail for unexpected reasons in the cluster, but when you "resume" it works again. To run the second workflow, I need that all the tasks from the first workflow are completed. So probably I will have to run "resume" several times on the first one.

My question is...how can I do it to automatically start the second workflow once all the tasks of the first one have been completed successfully? If it's easier to do it with processes instead of workflows I can also join them.

Thanks in advance for your help!!

[mribeirodantas]

If you can work at the level of processes, then: https://nextflow-io.github.io/patterns/process-collect/. If there isn't a channel dependency between this next process that needs all else to be finished before it starts, you can create a state dependency (read more at https://nextflow-io.github.io/patterns/state-dependency/) and use collect to only start when all tasks are finished.

[ericart89]

Thank you very much for your answer! But I just have a question...the fact that I use the "ignore" strategy in the first process (or workflow), would not make the second process take only those that succeed and ignore the others? I would like that the workflow stops here unless they all succeed. I know I could use "finish" or "terminate", but this would slow down the pipeline. Do you think there is any other alternative?

[mribeirodantas]

If you know beforehand the number of elements that you should receive at process 2, you can use the when block in the process to only instantiate the process if the number of elements is the one you expect. Or you can pass this through a map to the process and decide to interrupt the workflow if it's different from what you expected.

[ericart89]

Thanks I will do that!!!

[bentsherman]

You should be able to keep your workflows as they are and block until all samples have succeeded. You can use the collect operator to collect all the samples from workflow A, count to count the number of inputs from the input channel, combine to combine the input count with the processed samples, and filter to determine whether there are enough processed samples.

[bentsherman]

As for how to put them together... I will leave that as an exercise for the reader 😉