Merge branch 'master' into fix-azure-starttask-concatenation

Author: adamrtalbot

build.gradle

@@ -102,8 +102,8 @@ allprojects {
 
         // Documentation required libraries
         groovyDoc 'org.fusesource.jansi:jansi:2.4.0'
-        groovyDoc "org.apache.groovy:groovy-groovydoc:4.0.27"
-        groovyDoc "org.apache.groovy:groovy-ant:4.0.27"
+        groovyDoc "org.apache.groovy:groovy-groovydoc:4.0.28"
+        groovyDoc "org.apache.groovy:groovy-ant:4.0.28"
     }
 
     test {

docs/aws.md

@@ -281,7 +281,7 @@ There are several reasons why you might need to create your own [AMI (Amazon Mac
 ### Create your custom AMI
 
 From the EC2 Dashboard, select **Launch Instance**, then select **Browse more AMIs**. In the new page, select
-**AWS Marketplace AMIs**, and then search for **Amazon ECS-Optimized Amazon Linux 2 (AL2) x86_64 AMI**. Select the AMI and continue as usual to configure and launch the instance.
+**AWS Marketplace AMIs**, and then search for `Amazon ECS-Optimized Amazon Linux 2 (AL2) x86_64 AMI`. Select the AMI and continue as usual to configure and launch the instance.
 
 :::{note}
 The selected instance has a root volume of 30GB. Make sure to increase its size or add a second EBS volume with enough storage for real genomic workloads.

docs/cache-and-resume.md

@@ -72,75 +72,86 @@ For this reason, it is important to preserve both the task cache (`.nextflow/cac
 
 ## Troubleshooting
 
-Cache failures happen when either (1) a task that was supposed to be cached was re-executed, or (2) a task that was supposed to be re-executed was cached.
+Cache failures occur when a task that was supposed to be cached was re-executed or a task that was supposed to be re-executed was cached.
 
-When this happens, consider the following questions:
+Common causes of cache failures include:
 
-- Is resume enabled via `-resume`?
-- Is the {ref}`process-cache` directive set to a non-default value?
-- Is the task still present in the task cache and work directory?
-- Were any of the task inputs changed?
+- [Resume not enabled](#resume-not-enabled)
+- [Cache directive disabled](#cache-directive-disabled)
+- [Modified inputs](#modified-inputs)
+- [Inconsistent file attributes](#inconsistent-file-attributes)
+- [Race condition on a global variable](#race-condition-on-a-global-variable)
+- [Non-deterministic process inputs](#non-deterministic-process-inputs)
 
-Changing any of the inputs included in the [task hash](#task-hash) will invalidate the cache, for example:
+### Resume not enabled
 
-- Resuming from a different session ID
-- Changing the process name
-- Changing the task container image or Conda environment
-- Changing the task script
-- Changing an input file or bundled script used by the task
+The `-resume` option is required to resume a pipeline. Ensure you enable `-resume` in your run command or your Nextflow configuration file.
+
+### Cache directive disabled
 
-While the following examples would not invalidate the cache:
+The `cache` directive is enabled by default. However, you can disable or modify its behavior for a specific process. For example:
 
-- Changing the value of a directive (other than {ref}`process-ext`), even if that directive is used in the task script
+```nextflow
+process FOO {
+  cache false
+  // ...
+}
+```
 
-In many cases, cache failures happen because of a change to the pipeline script or configuration, or because the pipeline itself has some non-deterministic behavior.
+Ensure that the `cache` directive has not been disabled. See {ref}`process-cache` for more information.
 
-Here are some common reasons for cache failures:
+### Modified inputs
 
-### Modified input files
+Modifying inputs that are used in the task hash invalidates the cache. Common causes of modified inputs include:
 
-Make sure that your input files have not been changed. Keep in mind that the default caching mode uses the complete file path, the last modified timestamp, and the file size. If any of these attributes change, the task will be re-executed, even if the file content is unchanged.
+- Changing input files
+- Resuming from a different session ID
+- Changing the process name
+- Changing the calling workflow name
+- Changing the task container image or Conda environment
+- Changing the task script
+- Changing a bundled script used by the task
 
-### Process that modifies its inputs
+Nextflow calculates a hash for an input file using its full path, last modified timestamp, and file size. If any of these attributes change, Nextflow re-executes the task.
 
-If a process modifies its own input files, it cannot be resumed for the reasons described in the previous point. As a result, processes that modify their own input files are considered an anti-pattern and should be avoided.
+:::{warning}
+If a process modifies its input files, it cannot be resumed. Avoid processes that modify their own input files as this is considered an anti-pattern.
+:::
 
 ### Inconsistent file attributes
 
-Some shared file systems, such as NFS, may report inconsistent file timestamps, which can invalidate the cache. If you encounter this problem, you can avoid it by using the `'lenient'` {ref}`caching mode <process-cache>`, which ignores the last modified timestamp and uses only the file path and size.
+Some shared file systems, such as NFS, may report inconsistent file timestamps, which can invalidate the cache when using the standard caching mode.
+
+To resolve this issue, use the `'lenient'` {ref}`caching mode <process-cache>` to ignore the last modified timestamp and use only the file path and size.
 
 (cache-global-var-race-condition)=
 
 ### Race condition on a global variable
 
-While Nextflow tries to make it easy to write safe concurrent code, it is still possible to create race conditions, which can in turn impact the caching behavior of your pipeline.
-
-Consider the following example:
+Race conditions can disrupt the caching behavior of your pipeline. For example:
 
 ```nextflow
-channel.of(1,2,3) | map { v -> X=v; X+=2 } | view { v -> "ch1 = $v" }
-channel.of(1,2,3) | map { v -> X=v; X*=2 } | view { v -> "ch2 = $v" }
+channel.of(1,2,3).map { v -> X=v; X+=2 }.view { v -> "ch1 = $v" }
+channel.of(1,2,3).map { v -> X=v; X*=2 }.view { v -> "ch2 = $v" }
 ```
 
-The problem here is that `X` is declared in each `map` closure without the `def` keyword (or other type qualifier). Using the `def` keyword makes the variable local to the enclosing scope; omitting the `def` keyword makes the variable global to the entire script.
+In the above example, `X` is declared in each `map` closure. Without the `def` keyword, the variable `X` is global to the entire script. Because operators are executed concurrently and `X` is global, there is a *race condition* that causes the emitted values to vary depending on the order of the concurrent operations. If these values were passed to a process as inputs, the process would execute different tasks during each run due to the race condition.
 
-Because `X` is global, and operators are executed concurrently, there is a *race condition* on `X`, which means that the emitted values will vary depending on the particular order of the concurrent operations. If the values were passed as inputs into a process, the process would execute different tasks on each run due to the race condition.
-
-The solution is to not use a global variable where a local variable is enough (or in this simple example, avoid the variable altogether):
+To resolve this issue, avoid declaring global variables in closures:
 
 ```nextflow
-// local variable
-channel.of(1,2,3) | map { v -> def X=v; X+=2 } | view { v -> "ch1 = $v" }
-
-// no variable
-channel.of(1,2,3) | map { v -> v * 2 } | view { v -> "ch2 = $v" }
+channel.of(1,2,3).map { v -> def X=v; X+=2 }.view { v -> "ch1 = $v" }
 ```
 
+:::{versionadded} 25.04.0
+The {ref}`strict syntax <strict-syntax-page>` does not allow global variables to be declared in closures.
+:::
+
 (cache-nondeterministic-inputs)=
 
 ### Non-deterministic process inputs
 
-Sometimes a process needs to merge inputs from different sources. Consider the following example:
+A process that merges inputs from different sources non-deterministically may invalidate the cache. For example:
 
 ```nextflow
 workflow {
@@ -161,9 +172,9 @@ process check_bam_bai {
 }
 ```
 
-It is tempting to assume that the process inputs will be matched by `id` like the {ref}`operator-join` operator. But in reality, they are simply merged like the {ref}`operator-merge` operator. As a result, not only will the process inputs be incorrect, they will also be non-deterministic, thus invalidating the cache.
+In the above example, the inputs will be merged without matching on `id`, in a similar manner as the {ref}`operator-merge` operator. As a result, the inputs are incorrect and non-deterministic.
 
-The solution is to explicitly join the two channels before the process invocation:
+To resolve this issue, use the `join` operator to join the channels into a single input channel before invoking the process:
 
 ```nextflow
 workflow {

docs/conf.py

@@ -51,7 +51,11 @@
     'operator.md': 'reference/operator.md',
     'dsl1.md': 'migrations/dsl1.md',
     'updating-syntax.md': 'strict-syntax.md',
-    'updating-spot-retries.md': 'guides/updating-spot-retries.md'
+    'updating-spot-retries.md': 'guides/updating-spot-retries.md',
+    'metrics.md': 'tutorials/metrics.md',
+    'data-lineage.md' : 'tutorials/data-lineage.md',
+    'workflow-outputs.md': 'tutorials/workflow-outputs.md',
+    'flux.md': 'tutorials/flux.md'
 }
 
 # Add any paths that contain templates here, relative to this directory.

docs/developer-env.md

@@ -96,9 +96,9 @@ See {ref}`vscode-page` for more information about the Nextflow extension feature
 
 **nf-core**
 
-The [nf-core extension pack](https://marketplace.visualstudio.com/items?itemName=nf-core.nf-core-extensionpack) adds a selection of tools that help develop with nf-core, a community effort to collect a curated set of analysis pipelines built using Nextflow.
+[nf-core](https://nf-co.re/) is a community effort to collect a curated set of analysis pipelines built using Nextflow. The [nf-core extension pack](https://marketplace.visualstudio.com/items?itemName=nf-core.nf-core-extensionpack) adds a selection of tools that support development. For example, it includes [Code Spell Checker](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker), [Prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode), [Todo Tree](https://marketplace.visualstudio.com/items?itemName=Gruntfuggly.todo-tree), and [Markdown Extended](https://marketplace.visualstudio.com/items?itemName=jebbs.markdown-extended).
 
-The nf-core extension pack includes several useful extensions. For example, [Code Spell Checker](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker), [Prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode), [Todo Tree](https://marketplace.visualstudio.com/items?itemName=Gruntfuggly.todo-tree), and [Markdown Extended](https://marketplace.visualstudio.com/items?itemName=jebbs.markdown-extended). See [nf-core extension pack](https://marketplace.visualstudio.com/items?itemName=nf-core.nf-core-extensionpack) for more information about the tools included in the nf-core extension pack.
+See the [nf-core extension pack](https://marketplace.visualstudio.com/items?itemName=nf-core.nf-core-extensionpack) for more information about the included tools.
 
 (devenv-remote)=
 

docs/guides/aws-java-sdk-v2.md

@@ -2,13 +2,13 @@
 
 # AWS Java SDK v2
 
-AWS Java SDK v1 is reaching end of life at the end of 2025. Starting in version `25.06.0-edge`, Nextflow uses AWS Java SDK v2 in the `nf-amazon` plugin.
+AWS Java SDK v1 will reach end of life at the end of 2025. Starting with version `25.06.0-edge`, Nextflow uses AWS Java SDK v2 in the `nf-amazon` plugin.
 
-This migration introduced several breaking changes to the `aws.client` config scope, including new options and removed options. This page describes these changes and how they affect your Nextflow configuraiton.
+This migration introduces several breaking changes to the `aws.client` config scope, including new and removed options. This page describes these changes and how they affect your Nextflow configuration.
 
 ## New HTTP client
 
-The HTTP client used by SDK v2 does not support overriding certain advanced HTTP options. As a result, the following config options are no longer supported:
+The HTTP client in SDK v2 does not support overriding certain advanced HTTP options. As a result, the following config options are no longer supported:
 
 - `aws.client.protocol`
 - `aws.client.signerOverride`
@@ -18,15 +18,15 @@ The HTTP client used by SDK v2 does not support overriding certain advanced HTTP
 
 ## S3 transfer manager
 
-The *S3 transfer manager* is a subsystem of SDK v2 which handles S3 transfers, including S3 uploads and downloads.
+The *S3 transfer manager* is a subsystem of SDK v2 that handles S3 uploads and downloads.
 
-The concurrency and throughput of the S3 transfer manager can be configured manually using the `aws.client.maxConcurrency` and `aws.client.maxNativeMemory` config options. Alternatively, the `aws.client.targetThroughputInGbps` config option can be used to set the previous two options automatically based on a target throughput.
+You can configure the concurrency and throughput of the S3 transfer manager manually using the `aws.client.maxConcurrency` and `aws.client.maxNativeMemory` configuration options. Alternatively, you can use the `aws.client.targetThroughputInGbps` option to set both values automatically based on a target throughput.
 
-## Multi-part uplaods
+## Multi-part uploads
 
-Multi-part uploads are handled by the S3 transfer manager. The `aws.client.minimumPartSize` and `aws.client.multipartThreshold` config options can be used to control when and how multi-part uploads are performed.
+Multi-part uploads are handled by the S3 transfer manager. You can use the `aws.client.minimumPartSize` and `aws.client.multipartThreshold` config options to control when and how multi-part uploads are performed.
 
-The following multi-part upload options are no longer supported:
+The following multi-part upload config options are no longer supported:
 
 - `aws.client.uploadChunkSize`
 - `aws.client.uploadMaxAttempts`

docs/migrations/24-04.md

@@ -112,7 +112,7 @@ The `nf-ga4gh` plugin has since been moved into its own repository, [nextflow-io
   ```groovy
   conda.channels = ['seqera', 'conda-forge', 'bioconda', 'defaults']
 
-## Miscellanous
+## Miscellaneous
 
 - New config option: `azure.batch.pools.<name>.lowPriority`
 - New config option: `azure.batch.pools.<name>.startTask.script`

docs/migrations/24-10.md

@@ -35,7 +35,7 @@ Nextflow now supports managed identities for the Azure Batch executor. See {ref}
 
 <h3>Task previous execution trace</h3>
 
-The `task` variable in the process definition has two new proprties, `task.previousTrace` and `task.previousException`, which allows a task to access the runtime metadata of the previous attempt. See {ref}`task-previous-execution-trace` for details.
+The `task` variable in the process definition has two new properties, `task.previousTrace` and `task.previousException`, which allows a task to access the runtime metadata of the previous attempt. See {ref}`task-previous-execution-trace` for details.
 
 ## Breaking changes
 
@@ -53,7 +53,7 @@ The `task` variable in the process definition has two new proprties, `task.previ
 
 - The use of `addParams` and `params` clauses in include declarations is deprecated. See {ref}`module-params` for details.
 
-## Miscellanous
+## Miscellaneous
 
 - New config option: `aws.client.requesterPays`
 - New config option: `google.batch.autoRetryExitCodes`

docs/migrations/25-04.md

@@ -30,7 +30,7 @@ The third preview of workflow outputs introduces the following breaking changes
 
 - The syntax for dynamic publish paths has changed. Instead of defining a closure that returns a closure with the `path` directive, the outer closure should use the `>>` operator to publish individual files. See {ref}`workflow-publishing-files` for details.
 
-- The `mapper` index directive has been removed. Use a `map` operator in the workflwo body instead.
+- The `mapper` index directive has been removed. Use a `map` operator in the workflow body instead.
 
 See {ref}`migrating-workflow-outputs` to get started.
 

docs/reports.md

@@ -302,7 +302,7 @@ The following table shows the fields that can be included in the execution repor
 : The value of the process `scratch` directive.
 
 `error_action`
-: The action applied on errof task failure.
+: The action applied on error for task failure.
 
 `hostname`
 : :::{versionadded} 22.05.0-edge