How to make processes run sequentially in DSL2

[CB1915]

I have three processes and want them to be completed on after the other, what would be a good way to accomplish this?

Process1st {
     input: 
          val channelx
     output:
          stdout
     script:
          """
          Foo1 channelx
          """
}
Process2nd {
     input: 
          val channelx
     output:
          stdout
     script:
          """
          Foo2 channelx
          """
}
Process3rd {
     input: 
          val channelx
     output:
          stdout
     script:
          """
          Foo3 channelx
          """
}

channelx=channel.of(x)

workflow {
      Process1st(channelx)
      Process2nd(channelx)
      Process3rd(channelx)
}

[mribeirodantas]

According to your workflow block, there is no data dependency between the processes, which means there shouldn't be a reason for you to want them to be run sequentially. However, if you think there's still a need for them to be run sequentially, you can create state dependency as outlined here.

[colindaven]

Another way of answering would be, in code, imagining Process2nd takes the output from Process1st. Then Process2nd would wait for Process1st to complete before starting. You'd need to adjust the input for Process2nd as well potentially.

// use this in workflow block
Process2nd(Process1st.out)

[mribeirodantas]

It's basically what I said in my answer, @CB1915 😄. You create channels with meaningless elements to tie the processes together. So, they will run sequentially. If you want not only running sequentially but also only after each process has fully finished, you can also use the collect channel operator. Check the link I shared above in my answer and it'll become clear to you 😉

[CB1915]

@mribeirodantas I think I figured out how to apply what you said, thank you!

[mribeirodantas]

🥳

[CB1915]

@mribeirodantas so a slightly different example that I'm having some difficulty with (I guess I didn't understand as well as I thought)

#!/usr/bin/env nextflow

nextflow.enable.dsl=2

reads_ch=Channel.fromPath('/some/path/to/filepair{1,2}.fastq.gz')
workflow {
  doMove(reads_ch)
  doUnzip(doMove.out)
}

process doMove {
  input: 
    path reads_ch
    
  output:
    path(moved)
  script:
    
    
    """
    cp '${filepairs}' /some/new/path/
    """
    
}

process doUnzip {
  input: 
    path(moved)
    
  output:
    stdout
  script:
    
    
    """
    gunzip $moved
    """
    
}

this code pops out an error that is caused by Missing output file(s) moved expected by process doMove (21)

Do you have any advice for that?

[mribeirodantas]

Sure. First, this suggests that the way Nextflow works is not clear to you yet. I recommend you take this introductory course on Nextflow. In the meantime, I'll explain what's happening here.

You must specify what you want Nextflow to consider as output for a process if there's output. In doMove, you say the output is moved, but what's moved? 😅 The issue doesn't stop there either. In 99.9% of the time, you don't want to handle absolute paths in Nextflow, as Nextflow does that for you. Every task should be contained in its task directory, so this moving around is not required. Nextflow is the one that will move files in and out for you.

Very early in the course, as I mentioned above, you'll see many examples showing you how things should be done, e.g.:

process FOO {
  input:
  path input_file

  output:
  path 'output_file.txt'

  script:
  """
  cat ${input_file} > output_file.txt
  """
}

workflow {
  Channel
    .fromPath("*.txt")
    | FOO
}