Add progression

Author: pinin4fjords

docs/side_quests/workflows_of_workflows.md

@@ -48,7 +48,7 @@ modules/
 └── reverse_text.nf          # Reverses text content
 ```
 
-We going to compose these modules into two separate workflows that we will then compose into a main workflow.
+We're going to compose these modules into two separate workflows that we will then compose into a main workflow.
 
 ---
 
@@ -72,6 +72,50 @@ include { VALIDATE_NAME } from '../modules/validate_name'
 include { SAY_HELLO } from '../modules/say_hello'
 include { TIMESTAMP_GREETING } from '../modules/timestamp_greeting'
 
+workflow {
+
+    names_ch = Channel.from('Alice', 'Bob', 'Charlie')
+
+    // Chain processes: validate -> create greeting -> add timestamp
+    validated_ch = VALIDATE_NAME(names_ch)
+    greetings_ch = SAY_HELLO(validated_ch)
+    timestamped_ch = TIMESTAMP_GREETING(greetings_ch)
+}
+```
+
+This is a complete workflow, with a structure similar to the ones you saw in the 'Hello Nextflow' tutorial, that we can test independently. Let's try that now:
+
+```bash title="Run the greeting workflow"
+nextflow run workflows/greeting.nf
+```
+
+```console title="Expected output"
+N E X T F L O W  ~  version 24.10.0
+Launching `workflows/greeting.nf` [peaceful_montalcini] DSL2 - revision: 90f61b7093
+executor >  local (9)
+[51/4f980f] process > VALIDATE_NAME (validating Bob)                    [100%] 3 of 3 ✔
+[2b/dd8dc2] process > SAY_HELLO (greeting Bob)                          [100%] 3 of 3 ✔
+[8e/882565] process > TIMESTAMP_GREETING (adding timestamp to greeting) [100%] 3 of 3 ✔
+```
+
+This works as expected, but to make it composable there's a few things we need to change.
+
+### 1.3. Make the workflow composable
+
+Composable workflows have some differences from the ones you saw in the 'Hello Nextflow' tutorial:
+
+- The workflow block needs to be named
+- Inputs are declared using the `take:` keyword
+- Workflow content is placed inside the `main:` block
+- Outputs are declared using the `emit:` keyword
+
+Let's update the greeting workflow to match this structure. Change the code to the following:
+
+```groovy title="workflows/greeting.nf" linenums="1"
+include { VALIDATE_NAME } from '../modules/validate_name'
+include { SAY_HELLO } from '../modules/say_hello'
+include { TIMESTAMP_GREETING } from '../modules/timestamp_greeting'
+
 workflow GREETING_WORKFLOW {
     take:
         names_ch        // Input channel with names
@@ -88,9 +132,56 @@ workflow GREETING_WORKFLOW {
 }
 ```
 
-This is already a complete workflow that we can test independently. You might notice that it's very similar to the `hello` workflow from the 'Hello Nextflow' tutorial, but with some additional syntax to allow it to receive input ('take:') and produce output ('emit:'). These are the connections we will use to compose a higher level workflow.
+You can see that the workflow is now named and has a `take:` and `emit:` block, and these are the connections we will use to compose a higher level workflow.
+The workflow content is also placed inside the `main:` block. Note also that we have removed the `names_ch` input channel declaration, as it's now passed as an argument to the workflow.
+
+Let's test the workflow again to see if it works as expected:
+
+```bash title="Run the greeting workflow"
+nextflow run workflows/greeting.nf
+```
+
+What you'll actually see in response is:
+
+```console title="Expected output"
+N E X T F L O W  ~  version 24.10.0
+Launching `workflows/greeting.nf` [high_brahmagupta] DSL2 - revision: 8f5857af25
+WARN: No entry workflow specified
+```
+
+This tells you about another new concept, an 'entry workflow'. The entry workflow is the workflow that gets called when you run a Nextflow script. By default, Nextflow will use an un-named workflow as the entry workflow, when present, and that's what you've been doing so far, with workflow blocks starting like this:
+
+```groovy title="hello.nf" linenums="1"
+workflow {
+```
+
+But our greeting workflow doesn't have an un-named workflow, rather we have a named workflow:
+
+```groovy title="workflows/greeting.nf" linenums="1"
+workflow GREETING_WORKFLOW {
+```
+
+... so Nextflow will throw an error. We can actually tell Nextflow to use our named workflow as the entry workflow by adding this line to Nextflow's command line:
+
+```bash title="Run the greeting workflow"
+nextflow run workflows/greeting.nf -entry GREETING_WORKFLOW
+```
+
+This will also throw an error, because the workflow is expecting an input channel:
+
+```console title="Expected output"
+N E X T F L O W  ~  version 24.10.0
+Launching `workflows/greeting.nf` [compassionate_fermi] DSL2 - revision: 8f5857af25
+ERROR ~ Workflow `GREETING_WORKFLOW` declares 1 input channels but 0 were given
+
+ -- Check '.nextflow.log' file for details
+```
+
+... but if you wanted to call a named workflow that didn't require inputs, you could call it this way.
+
+But we didn't add that syntax so we could call the workflow directly, we did it so we could compose it with other workflows. Let's start by creating a main workflow that imports and uses the `greeting` workflow.
 
-### 1.3. Create and test the main workflow
+### 1.4. Create and test the main workflow
 
 Now we will create a main workflow that imports and uses the `greeting` workflow.
 
@@ -109,14 +200,16 @@ workflow {
 
 ```
 
+Note that our workflow entry in this file is un-named, and that's because we're going to use it as an entry workflow.
+
 Run this and see the output:
 
 ```bash title="Run the workflow"
 nextflow run main.nf
 ```
 
 ```console title="Expected output"
-N E X T F L O W  ~  version 23.10.1
+N E X T F L O W  ~  version 24.10.0
 Launching `main.nf` [goofy_mayer] DSL2 - revision: 543f8742fe
 executor >  local (9)
 [05/3cc752] process > GREETING_WORKFLOW:VALIDATE_NAME (validating Char... [100%] 3 of 3 ✔
@@ -130,14 +223,27 @@ Timestamped: /workspaces/training/side_quests/workflows_of_workflows/work/aa/bd3
 Timestamped: /workspaces/training/side_quests/workflows_of_workflows/work/ea/342168d4ba04cc899a89c56cbfd9b0/timestamped_Charlie-output.txt
 ```
 
+It works! We've wrapped the named greeting workflow in a main workflow with an un-named entry `workflow` block. The main workflow is using the `GREETING_WORKFLOW` workflow almost (not quite) like a process, and passing the `names` channel as an argument.
+
 ### Takeaway
 
-You should now have a working greeting workflow that:
+In this section, you've learned several important concepts:
+
+- **Named Workflows**: Creating a named workflow (`GREETING_WORKFLOW`) that can be imported and reused
+- **Workflow Interfaces**: Defining clear inputs with `take:` and outputs with `emit:` to create a composable workflow
+- **Entry Points**: Understanding that Nextflow needs an entry workflow (either unnamed or specified with `-entry`)
+- **Workflow Composition**: Importing and using a named workflow within another workflow
+- **Workflow Namespaces**: Accessing workflow outputs using the `.out` namespace (`GREETING_WORKFLOW.out.greetings`)
+
+You now have a working greeting workflow that:
 
 - Takes a channel of names as input
 - Validates each name
 - Creates a greeting for each valid name
 - Adds timestamps to the greetings
+- Exposes both original and timestamped greetings as outputs
+
+This modular approach allows you to test the greeting workflow independently or use it as a component in larger pipelines.
 
 ---
 
@@ -174,26 +280,28 @@ workflow TRANSFORM_WORKFLOW {
 }
 ```
 
+We won't repeat the explanation of the composable syntax here, but note the named workflow is again declared with a `take:` and `emit:` block, and the workflow content is placed inside the `main:` block.
+
 ### 2.3. Update the main workflow
 
 Update `main.nf` to use both workflows:
 
 ```groovy title="main.nf" linenums="1"
-include { SAY_HELLO_UPPER } from '../modules/say_hello_upper'
-include { REVERSE_TEXT } from '../modules/reverse_text'
+include { GREETING_WORKFLOW } from './workflows/greeting'
+include { TRANSFORM_WORKFLOW } from './workflows/transform'
 
-workflow TRANSFORM_WORKFLOW {
-    take:
-        input_ch         // Input channel with messages
+workflow {
+    names = Channel.from('Alice', 'Bob', 'Charlie')
 
-    main:
-        // Apply transformations in sequence
-        upper_ch = SAY_HELLO_UPPER(input_ch)
-        reversed_ch = REVERSE_TEXT(upper_ch)
+    // Run the greeting workflow
+    GREETING_WORKFLOW(names)
 
-    emit:
-        upper = upper_ch        // Uppercase greetings
-        reversed = reversed_ch  // Reversed uppercase greetings
+    // Run the transform workflow
+    TRANSFORM_WORKFLOW(GREETING_WORKFLOW.out.timestamped)
+
+    // View results
+    TRANSFORM_WORKFLOW.out.upper.view { "Uppercase: $it" }
+    TRANSFORM_WORKFLOW.out.reversed.view { "Reversed: $it" }
 }
 ```
 
@@ -204,7 +312,7 @@ nextflow run main.nf
 ```
 
 ```console title="Expected output"
-N E X T F L O W  ~  version 23.10.1
+N E X T F L O W  ~  version 24.10.0
 Launching `main.nf` [sick_kimura] DSL2 - revision: 8dc45fc6a8
 executor >  local (13)
 executor >  local (15)
@@ -223,12 +331,13 @@ Reversed: /workspaces/training/side_quests/workflows_of_workflows/work/f0/74ba4a
 
 If you take a look at one of those reversed files, you'll see that it's the uppercase version of the greeting reversed:
 
-````bash title="Check a final output file"
-cat /workspaces/training/side_quests/workflows_of_workflows/work/f0/74ba4a10d9ef5c82f829d1c154d0f6/REVERSED-UPPER-timestamped_Alice-output.txt```
+```bash title="Check a final output file"
+cat /workspaces/training/side_quests/workflows_of_workflows/work/f0/74ba4a10d9ef5c82f829d1c154d0f6/REVERSED-UPPER-timestamped_Alice-output.txt
+```
 
 ```console title="Reversed file content"
 !ECILA ,OLLEH ]04:50:71 60-30-5202[
-````
+```
 
 ### Takeaway
 
@@ -255,14 +364,19 @@ In this side quest, we've explored the powerful concept of workflow composition
 
 5. **Practiced Modular Design**: We experienced firsthand how breaking a pipeline into logical components makes the code more maintainable and easier to understand.
 
+6. **Worked with Entry Points**: We learned that Nextflow requires an entry workflow (either unnamed or specified with `-entry`) to know where to start execution.
+
+7. **Structured Workflow Content**: We wrapped workflow logic within the `main:` block.
+
 This modular approach offers several advantages over monolithic pipelines:
 
 - Each workflow can be developed, tested, and debugged independently
 - Workflows can be reused across different pipelines
 - The overall pipeline structure becomes more readable and maintainable
 - Changes to one workflow don't necessarily affect others if the interfaces remain consistent
+- Entry points can be configured to run different parts of your pipeline as needed
 
-It's important to note that while calling workflows is a bit like calling processes, it's not the same. You can't, for example, run a workflow n times by calling it with a channel of size n- you would need to pass a channel of size n to the workflow and iterate internally.
+It's important to note that while calling workflows is a bit like calling processes, it's not the same. You can't, for example, run a workflow n times by calling it with a channel of size n - you would need to pass a channel of size n to the workflow and iterate internally.
 
 By mastering workflow composition, you're now equipped to build more sophisticated Nextflow pipelines that can handle complex bioinformatics tasks while remaining maintainable and scalable.
 
@@ -292,13 +406,50 @@ By mastering workflow composition, you're now equipped to build more sophisticat
    }
    ```
 
-3. **Workflow Composition**
+3. **Main Block Structure**
+
+   ```nextflow
+   workflow EXAMPLE_WORKFLOW {
+       take:
+           // Input channels are declared here
+           input_ch
+
+       main:
+           // Workflow logic goes here
+           // This is where processes are called and channels are manipulated
+           result_ch = SOME_PROCESS(input_ch)
+
+       emit:
+           // Output channels are declared here
+           output_ch = result_ch
+   }
+   ```
+
+4. **Workflow Composition**
+
    ```nextflow
    // Using explicit connections
    WORKFLOW_A(input_ch)
    WORKFLOW_B(WORKFLOW_A.out.some_channel)
    ```
 
+5. **Entry Points**
+
+   ```nextflow
+   // Unnamed workflow (default entry point)
+   workflow {
+       // This is automatically the entry point when the script is run
+   }
+
+   // Named workflow (not an entry point by default)
+   workflow NAMED_WORKFLOW {
+       // This is not automatically run
+   }
+
+   // Running a named workflow as entry point
+   // nextflow run script.nf -entry NAMED_WORKFLOW
+   ```
+
 ## Resources
 
 - [Nextflow Workflow Documentation](https://www.nextflow.io/docs/latest/workflow.html)