Add Scripting essentials module (#652)

pinin4fjords · Copilot · claude · web-flow · commit 12d8cde9eafa · 2025-10-15T11:54:17.000+01:00
* Add AI-assisted Groovy essentials * Refine section 1.1 * Add groovy essentials to nav * Fix list rendering * Emphasise departure to groovy * Shorten intro * Shorten intro * Fix highlight * Refine section 1.2 * messy updates up to section 4 * State after a good chat wiht Claude code * Tone down intro * Reset collect.nf to starting state * Tweaks * Reset collect.nf to starting state * Tweaks * Revert main.nf to starting point * Tweaks * backticks fix? * Reset fastp module to starting point * Fix fastp * tweaks * latest tweaks, simplify the variable interpretation part * Reorg subsections a bit * Fix highlights * Fix up dynamic resources * Fix up dynamic routing * Final tweaks * Language/ clarity improvements * Prettier * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add teach time estimate for groovy * Fix formatting * Some section 2 fixes * Fix highlights * Fix tiny issues * More Groovy fixes * update time estimate * Refactor Groovy Essentials to Essential Scripting Patterns Address reviewer feedback to reframe content away from "Nextflow vs Groovy" and toward presenting Nextflow as a standalone language. Major changes: - Renamed from groovy_essentials to essential_scripting_patterns - Updated terminology: "dataflow vs scripting" instead of "Nextflow vs Groovy" - Reframed collect example as Channel vs Iterable types in Nextflow standard library - Added historical context about Groovy while emphasizing Nextflow language specification - Updated all code comments and examples to reference Nextflow standard library - Prioritized Nextflow documentation in resources section 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Prettier * Update title to Essential Nextflow Scripting Patterns 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Apply suggestions from code review Co-authored-by: Ben Sherman <bentshermann@gmail.com> * Easy fixes for Ben * Iterable -> List * Fixes up to handlers * Fix highlight * Tighten intro * Tiny fixes --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Ben Sherman <bentshermann@gmail.com>
diff --git a/docs/side_quests/essential_scripting_patterns.md b/docs/side_quests/essential_scripting_patterns.md
diff --git a/docs/side_quests/index.md b/docs/side_quests/index.md
@@ -28,15 +28,16 @@ Otherwise, select a side quest from the table below.
 
 ## Side Quests
 
-| Side Quest                                                        | Time Estimate for Teaching |
-| ----------------------------------------------------------------- | -------------------------- |
-| [Nextflow development environment walkthrough](./ide_features.md) | 45 mins                    |
-| [Introduction to nf-core](./nf-core.md)                           | -                          |
-| [Metadata in workflows](./metadata.md)                            | 45 mins                    |
-| [Splitting and Grouping](./splitting_and_grouping.md)             | 45 mins                    |
-| [Testing with nf-test](./nf-test.md)                              | 1 hour                     |
-| [Workflows of workflows](./workflows_of_workflows.md)             | 30 mins                    |
-| [Working with files](./working_with_files.md)                     | 45 mins                    |
-| [Debugging workflows](./debugging.md)                             | 1 hour                     |
+| Side Quest                                                                 | Time Estimate for Teaching |
+| -------------------------------------------------------------------------- | -------------------------- |
+| [Nextflow development environment walkthrough](./ide_features.md)          | 45 mins                    |
+| [Essential Nextflow Scripting Patterns](./essential_scripting_patterns.md) | 90 mins                    |
+| [Introduction to nf-core](./nf-core.md)                                    | -                          |
+| [Metadata in workflows](./metadata.md)                                     | 45 mins                    |
+| [Splitting and Grouping](./splitting_and_grouping.md)                      | 45 mins                    |
+| [Testing with nf-test](./nf-test.md)                                       | 1 hour                     |
+| [Workflows of workflows](./workflows_of_workflows.md)                      | 30 mins                    |
+| [Working with files](./working_with_files.md)                              | 45 mins                    |
+| [Debugging workflows](./debugging.md)                                      | 1 hour                     |
 
 Let us know what other domains and use cases you'd like to see covered here by posting in the [Training section](https://community.seqera.io/c/training/) of the community forum.
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -65,6 +65,7 @@ nav:
       - side_quests/splitting_and_grouping.md
       - side_quests/workflows_of_workflows.md
       - side_quests/debugging.md
+      - side_quests/essential_scripting_patterns.md
       - side_quests/nf-test.md
       - side_quests/nf-core.md
   - Archive:
diff --git a/side-quests/essential_scripting_patterns/collect.nf b/side-quests/essential_scripting_patterns/collect.nf
@@ -0,0 +1,7 @@
+def sample_ids = ['sample_001', 'sample_002', 'sample_003']
+
+// channel.collect() - groups multiple channel emissions into one
+ch_input = channel.fromList(sample_ids)
+ch_input.view { sample -> "Individual channel item: ${sample}" }
+ch_collected = ch_input.collect()
+ch_collected.view { list -> "channel.collect() result: ${list} (${list.size()} items grouped into 1)" }
diff --git a/side-quests/essential_scripting_patterns/data/samples.csv b/side-quests/essential_scripting_patterns/data/samples.csv
@@ -0,0 +1,4 @@
+sample_id,organism,tissue_type,sequencing_depth,file_path,quality_score
+SAMPLE_001,human,liver,30000000,data/sequences/SAMPLE_001_S1_L001_R1_001.fastq,38.5
+SAMPLE_002,mouse,brain,25000000,data/sequences/SAMPLE_002_S2_L001_R1_001.fastq,35.2
+SAMPLE_003,human,kidney,45000000,data/sequences/SAMPLE_003_S3_L001_R1_001.fastq,42.1
diff --git a/side-quests/essential_scripting_patterns/data/sequences/SAMPLE_001_S1_L001_R1_001.fastq b/side-quests/essential_scripting_patterns/data/sequences/SAMPLE_001_S1_L001_R1_001.fastq
@@ -0,0 +1,12 @@
+@sample_001_read_1
+ATGCGATCGATCGATCGATCGATCGATCGATCGATCGATCGATC
++
+IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
+@sample_001_read_2
+GCATCGATCGATCGATCGATCGATCGATCGATCGATCGATCGAT
++
+HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
+@sample_001_read_3
+TCGATCGATCGATCGATCGATCGATCGATCGATCGATCGATCGA
++
+JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJH
diff --git a/side-quests/essential_scripting_patterns/data/sequences/SAMPLE_002_S2_L001_R1_001.fastq b/side-quests/essential_scripting_patterns/data/sequences/SAMPLE_002_S2_L001_R1_001.fastq
@@ -0,0 +1,12 @@
+@sample_002_read_1
+CGATCGATCGATCGATCGATCGATCGATCGATCGATCGATCGAT
++
+IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
+@sample_002_read_2
+ATCGATCGATCGATCGATCGATCGATCGATCGATCGATCGATCG
++
+HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
+@sample_002_read_3
+GATCGATCGATCGATCGATCGATCGATCGATCGATCGATCGATC
++
+JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
diff --git a/side-quests/essential_scripting_patterns/data/sequences/SAMPLE_003_S3_L001_R1_001.fastq b/side-quests/essential_scripting_patterns/data/sequences/SAMPLE_003_S3_L001_R1_001.fastq
@@ -0,0 +1,12 @@
+@sample_003_read_1
+GCGCGCGCGCGCGCGCGCGCGCGCGCGCGCGCGCGCGCGCGC
++
+IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
+@sample_003_read_2
+CGCGCGCGCGCGCGCGCGCGCGCGCGCGCGCGCGCGCGCGCG
++
+HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
+@sample_003_read_3
+ATATATATATATATATATATATATATATATATATATATAT
++
+JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
diff --git a/side-quests/essential_scripting_patterns/main.nf b/side-quests/essential_scripting_patterns/main.nf
@@ -0,0 +1,5 @@
+workflow {
+    ch_samples = channel.fromPath("./data/samples.csv")
+        .splitCsv(header: true)
+        .view()
+}
diff --git a/side-quests/essential_scripting_patterns/modules/fastp.nf b/side-quests/essential_scripting_patterns/modules/fastp.nf
@@ -0,0 +1,21 @@
+process FASTP {
+    container 'community.wave.seqera.io/library/fastp:0.24.0--62c97b06e8447690'
+
+    input:
+    tuple val(meta), path(reads)
+
+    output:
+    tuple val(meta), path("*_trimmed*.fastq.gz"), emit: reads
+
+    script:
+    """
+    fastp \\
+        --in1 ${reads[0]} \\
+        --in2 ${reads[1]} \\
+        --out1 ${meta.id}_trimmed_R1.fastq.gz \\
+        --out2 ${meta.id}_trimmed_R2.fastq.gz \\
+        --json ${meta.id}.fastp.json \\
+        --html ${meta.id}.fastp.html \\
+        --thread $task.cpus
+    """
+}
diff --git a/side-quests/essential_scripting_patterns/modules/generate_report.nf b/side-quests/essential_scripting_patterns/modules/generate_report.nf
@@ -0,0 +1,16 @@
+process GENERATE_REPORT {
+
+    publishDir 'results/reports', mode: 'copy'
+
+    input:
+    tuple val(meta), path(reads)
+
+    output:
+    path "${meta.id}_report.txt"
+
+    script:
+    """
+    echo "Processing ${reads}" > ${meta.id}_report.txt
+    echo "Sample: ${meta.id}" >> ${meta.id}_report.txt
+    """
+}
diff --git a/side-quests/essential_scripting_patterns/modules/trimgalore.nf b/side-quests/essential_scripting_patterns/modules/trimgalore.nf
@@ -0,0 +1,37 @@
+process TRIMGALORE {
+    container 'quay.io/biocontainers/trim-galore:0.6.10--hdfd78af_0'
+
+    input:
+    tuple val(meta), path(reads)
+
+    output:
+    tuple val(meta), path("*_trimmed*.fq"), emit: reads
+    path "*_trimming_report.txt"                   , emit: reports
+
+    script:
+    // Simple single-end vs paired-end detection
+    def is_single = reads instanceof List ? reads.size() == 1 : true
+
+    if (is_single) {
+        def input_file = reads instanceof List ? reads[0] : reads
+        """
+        trim_galore \\
+            --cores $task.cpus \\
+            ${input_file}
+
+        # Rename output to match expected pattern
+        mv *_trimmed.fq ${meta.id}_trimmed.fq
+        """
+    } else {
+        """
+        trim_galore \\
+            --paired \\
+            --cores $task.cpus \\
+            ${reads[0]} ${reads[1]}
+
+        # Rename outputs to match expected pattern
+        mv *_val_1.fq ${meta.id}_trimmed_R1.fq
+        mv *_val_2.fq ${meta.id}_trimmed_R2.fq
+        """
+    }
+}
diff --git a/side-quests/essential_scripting_patterns/nextflow.config b/side-quests/essential_scripting_patterns/nextflow.config
@@ -0,0 +1,3 @@
+// Nextflow configuration for Groovy Essentials side quest
+
+docker.enabled = true
diff --git a/side-quests/solutions/essential_scripting_patterns/collect.nf b/side-quests/solutions/essential_scripting_patterns/collect.nf
@@ -0,0 +1,18 @@
+def sample_ids = ['sample_001', 'sample_002', 'sample_003']
+
+// channel.collect() - groups multiple channel emissions into one
+ch_input = channel.fromList(sample_ids)
+ch_input.view { sample -> "Individual channel item: ${sample}" }
+ch_collected = ch_input.collect()
+ch_collected.view { list -> "channel.collect() result: ${list} (${list.size()} items grouped into 1)" }
+
+// List.collect() - transforms each element, preserves structure
+def formatted_ids = sample_ids.collect { id ->
+    id.toUpperCase().replace('SAMPLE_', 'SPECIMEN_')
+}
+println "List.collect() result: ${formatted_ids} (${sample_ids.size()} items transformed into ${formatted_ids.size()})"
+
+// Spread operator - concise property access
+def sample_data = [[id: 's1', quality: 38.5], [id: 's2', quality: 42.1], [id: 's3', quality: 35.2]]
+def all_ids = sample_data*.id
+println "Spread operator result: ${all_ids}"
diff --git a/side-quests/solutions/essential_scripting_patterns/main.nf b/side-quests/solutions/essential_scripting_patterns/main.nf
@@ -0,0 +1,88 @@
+include { FASTP } from './modules/fastp.nf'
+include { TRIMGALORE } from './modules/trimgalore.nf'
+include { GENERATE_REPORT } from './modules/generate_report.nf'
+
+def validateInputs() {
+    // Check input parameter is provided
+    if (!params.input) {
+        error("Input CSV file path not provided. Please specify --input <file.csv>")
+    }
+
+    // Check CSV file exists
+    if (!file(params.input).exists()) {
+        error("Input CSV file not found: ${params.input}")
+    }
+}
+
+def separateMetadata(row) {
+    def sample_meta = [
+        id: row.sample_id.toLowerCase(),
+        organism: row.organism,
+        tissue: row.tissue_type.replaceAll('_', ' ').toLowerCase(),
+        depth: row.sequencing_depth.toInteger(),
+        quality: row.quality_score?.toDouble()
+    ]
+    def run_id = row.run_id?.toUpperCase() ?: 'UNSPECIFIED'
+    sample_meta.run = run_id
+    def fastq_path = file(row.file_path)
+
+    def m = (fastq_path.name =~ /^(.+)_S(\d+)_L(\d{3})_(R[12])_(\d{3})\.fastq(?:\.gz)?$/)
+    def file_meta = m ? [
+        sample_num: m[0][2].toInteger(),
+        lane: m[0][3],
+        read: m[0][4],
+        chunk: m[0][5]
+    ] : [:]
+
+    def priority = sample_meta.quality > 40 ? 'high' : 'normal'
+
+    // Validate data makes sense
+    if (sample_meta.depth < 30000000) {
+        log.warn "Low sequencing depth for ${sample_meta.id}: ${sample_meta.depth}"
+    }
+
+    return tuple(sample_meta + file_meta + [priority: priority], fastq_path)
+}
+
+workflow {
+    validateInputs()
+
+    ch_samples = channel.fromPath(params.input)
+        .splitCsv(header: true)
+        .map{ row -> separateMetadata(row) }
+
+    // Filter out invalid or low-quality samples
+    ch_valid_samples = ch_samples
+        .filter { meta, reads ->
+            meta.id && meta.organism && meta.depth > 25000000
+        }
+
+    trim_branches = ch_valid_samples
+        .branch { meta, reads ->
+            fastp: meta.organism == 'human' && meta.depth >= 30000000
+            trimgalore: true
+        }
+
+    ch_fastp = FASTP(trim_branches.fastp)
+    ch_trimgalore = TRIMGALORE(trim_branches.trimgalore)
+    GENERATE_REPORT(ch_samples)
+
+    workflow.onComplete = {
+        println ""
+        println "Pipeline execution summary:"
+        println "=========================="
+        println "Completed at: ${workflow.complete}"
+        println "Duration    : ${workflow.duration}"
+        println "Success     : ${workflow.success}"
+        println "workDir     : ${workflow.workDir}"
+        println "exit status : ${workflow.exitStatus}"
+        println ""
+
+        if (workflow.success) {
+            println "✅ Pipeline completed successfully!"
+        } else {
+            println "❌ Pipeline failed!"
+            println "Error: ${workflow.errorMessage}"
+        }
+    }
+}
diff --git a/side-quests/solutions/essential_scripting_patterns/modules/fastp.nf b/side-quests/solutions/essential_scripting_patterns/modules/fastp.nf
@@ -0,0 +1,37 @@
+process FASTP {
+    container 'community.wave.seqera.io/library/fastp:0.24.0--62c97b06e8447690'
+
+    input:
+    tuple val(meta), path(reads)
+
+    output:
+    tuple val(meta), path("*_trimmed*.fastq.gz"), emit: reads
+    path "*.{json,html}"                             , emit: reports
+
+    script:
+    // Simple single-end vs paired-end detection
+    def is_single = reads instanceof List ? reads.size() == 1 : true
+
+    if (is_single) {
+        def input_file = reads instanceof List ? reads[0] : reads
+        """
+        fastp \\
+            --in1 ${input_file} \\
+            --out1 ${meta.id}_trimmed.fastq.gz \\
+            --json ${meta.id}.fastp.json \\
+            --html ${meta.id}.fastp.html \\
+            --thread $task.cpus
+        """
+    } else {
+        """
+        fastp \\
+            --in1 ${reads[0]} \\
+            --in2 ${reads[1]} \\
+            --out1 ${meta.id}_trimmed_R1.fastq.gz \\
+            --out2 ${meta.id}_trimmed_R2.fastq.gz \\
+            --json ${meta.id}.fastp.json \\
+            --html ${meta.id}.fastp.html \\
+            --thread $task.cpus
+        """
+    }
+}
diff --git a/side-quests/solutions/essential_scripting_patterns/modules/generate_report.nf b/side-quests/solutions/essential_scripting_patterns/modules/generate_report.nf
@@ -0,0 +1,19 @@
+process GENERATE_REPORT {
+
+    publishDir 'results/reports', mode: 'copy'
+
+    input:
+    tuple val(meta), path(reads)
+
+    output:
+    path "${meta.id}_report.txt"
+
+    script:
+    """
+    echo "Processing ${reads}" > ${meta.id}_report.txt
+    echo "Sample: ${meta.id}" >> ${meta.id}_report.txt
+    echo "Processed by: \${USER}" >> ${meta.id}_report.txt
+    echo "Hostname: \$(hostname)" >> ${meta.id}_report.txt
+    echo "Date: \$(date)" >> ${meta.id}_report.txt
+    """
+}
diff --git a/side-quests/solutions/essential_scripting_patterns/modules/trimgalore.nf b/side-quests/solutions/essential_scripting_patterns/modules/trimgalore.nf
@@ -0,0 +1,37 @@
+process TRIMGALORE {
+    container 'quay.io/biocontainers/trim-galore:0.6.10--hdfd78af_0'
+
+    input:
+    tuple val(meta), path(reads)
+
+    output:
+    tuple val(meta), path("*_trimmed*.fq"), emit: reads
+    path "*_trimming_report.txt"                   , emit: reports
+
+    script:
+    // Simple single-end vs paired-end detection
+    def is_single = reads instanceof List ? reads.size() == 1 : true
+
+    if (is_single) {
+        def input_file = reads instanceof List ? reads[0] : reads
+        """
+        trim_galore \\
+            --cores $task.cpus \\
+            ${input_file}
+
+        # Rename output to match expected pattern
+        mv *_trimmed.fq ${meta.id}_trimmed.fq
+        """
+    } else {
+        """
+        trim_galore \\
+            --paired \\
+            --cores $task.cpus \\
+            ${reads[0]} ${reads[1]}
+
+        # Rename outputs to match expected pattern
+        mv *_val_1.fq ${meta.id}_trimmed_R1.fq
+        mv *_val_2.fq ${meta.id}_trimmed_R2.fq
+        """
+    }
+}
diff --git a/side-quests/solutions/essential_scripting_patterns/nextflow.config b/side-quests/solutions/essential_scripting_patterns/nextflow.config
@@ -0,0 +1,3 @@
+// Nextflow configuration for Groovy Essentials side quest
+
+docker.enabled = true
