diff --git a/.docusaurus_site/README.md b/.docusaurus_site/README.md index 0e3c19d125..167df791f4 100644 --- a/.docusaurus_site/README.md +++ b/.docusaurus_site/README.md @@ -117,6 +117,10 @@ workflow { Supported languages include: `groovy`, `bash`, `python`, `javascript`, `java`, `yaml`, `json`, and many more. +### Runnable snippets + +- Code snippets under `docs/snippets/` are runnable and covered by the docs tests. + ### Images and assets - Place images in `docs/_static/` or `static/` diff --git a/.docusaurus_site/sidebars.js b/.docusaurus_site/sidebars.js index e9a0fc07e1..e6afcda711 100644 --- a/.docusaurus_site/sidebars.js +++ b/.docusaurus_site/sidebars.js @@ -32,9 +32,7 @@ module.exports = { "script", "working-with-files", "process", - "process-typed", "workflow", - "workflow-typed", "notifications", "secrets", "sharing", @@ -45,13 +43,34 @@ module.exports = { type: "category", label: "Modules", collapsed: true, + link: { type: "doc", id: "modules/modules" }, items: [ - "modules/modules", "modules/using-modules", "modules/developing-modules", "modules/module-registry" ] }, + { + type: "category", + label: "Static typing", + collapsed: true, + link: { type: "doc", id: "static-typing" }, + items: [ + "process-typed", + { + type: "category", + label: "Workflows (typed)", + collapsed: true, + link: { type: "doc", id: "workflow-typed" }, + items: [ + "typed-parameters", + "typed-outputs", + "typed-workflows" + ] + }, + "reference/operator-typed" + ] + }, { type: "category", label: "Software dependencies", @@ -108,7 +127,6 @@ module.exports = { }, "reference/process", "reference/channel", - "reference/operator-typed", "reference/operator" ] }, diff --git a/docs/README.md b/docs/README.md deleted file mode 100644 index f659e87a42..0000000000 --- a/docs/README.md +++ /dev/null @@ -1,22 +0,0 @@ -# Nextflow documentation - -This directory holds the Nextflow documentation content as `.mdx` files. The site is built with [Docusaurus](https://docusaurus.io/) and deployed to [docs.seqera.io/nextflow](https://docs.seqera.io/nextflow/) via Netlify. - -## Editing docs - -- Edit the `.mdx` files in this directory. -- Use relative links for internal references, e.g. `[link text](./other-page.mdx)`. -- Images and other assets live in `_static/`. -- Code snippets under `snippets/` are runnable and covered by the docs tests. - -## Building and previewing locally - -The Docusaurus project lives in [`.docusaurus_site/`](../.docusaurus_site) at the repository root (this directory is symlinked into it as `docs/`): - -```bash -cd .docusaurus_site -npm install -npm start -``` - -See [`.docusaurus_site/README.md`](../.docusaurus_site/README.md) for the full build, configuration, and authoring guide. diff --git a/docs/migrations/25-10.mdx b/docs/migrations/25-10.mdx index 3c286125f9..1a124d656e 100644 --- a/docs/migrations/25-10.mdx +++ b/docs/migrations/25-10.mdx @@ -307,6 +307,6 @@ This feature addresses previous inconsistencies in timestamp representations. [stdlib-types]: ../reference/stdlib-types [strict-syntax-page]: ../strict-syntax [workflow-output-def]: ../workflow#outputs -[workflow-typed-params]: ../workflow-typed#typed-parameters +[workflow-typed-params]: ../typed-parameters [workflow-params-legacy]: ../workflow#parameters [workflow-handlers]: ../notifications#workflow-handlers diff --git a/docs/migrations/26-04.mdx b/docs/migrations/26-04.mdx index 88a10805af..f8ae8390a6 100644 --- a/docs/migrations/26-04.mdx +++ b/docs/migrations/26-04.mdx @@ -443,4 +443,4 @@ A Platform access token with appropriate permissions is required to download non [stdlib-listDirectory]: ../reference/stdlib-types#listdirectory---iterablepath [strict-syntax-page]: ../strict-syntax [workflow-typed-page]: ../workflow-typed -[workflow-typed-params]: ../workflow-typed#typed-parameters +[workflow-typed-params]: ../typed-parameters diff --git a/docs/process-typed.mdx b/docs/process-typed.mdx index be961cdf32..364c010f7f 100644 --- a/docs/process-typed.mdx +++ b/docs/process-typed.mdx @@ -30,17 +30,8 @@ process hello { } ``` -To use this feature: - -1. Enable the [strict parser][strict-syntax-page] by setting the `NXF_SYNTAX_PARSER` environment variable to `v2`: - - ```bash - export NXF_SYNTAX_PARSER=v2 - ``` - -2. Set `nextflow.enable.types = true` in every script that uses typed processes. - -See [Process (typed)][syntax-process-typed] for the complete syntax reference and [Migrating to static typing][migrating-static-types] to migrate existing code to static typing. +For an overview of static typing and how to enable it, see [Static typing][static-typing-page]. +For the complete syntax reference, see [Process (typed)][syntax-process-typed]. ## Inputs @@ -239,7 +230,7 @@ process grep { } ``` -See [Inputs and outputs (typed)][process-reference-typed] for available stage directives. +For more information about the available stage directives, see [Inputs and outputs (typed)][process-reference-typed]. ## Outputs @@ -283,7 +274,7 @@ process echo { } ``` -See [Inputs and outputs (typed)][process-reference-typed] for available output functions. +For more information about the available output functions, see [Inputs and outputs (typed)][process-reference-typed]. ### File outputs @@ -388,6 +379,6 @@ Directives behave the same way as [legacy processes][process-directives]. [process-directives]: ./process#directives [process-script]: ./process#script [process-stub]: ./process#stub +[static-typing-page]: ./static-typing [stdlib-types]: ./reference/stdlib-types -[strict-syntax-page]: ./strict-syntax [syntax-process-typed]: ./reference/syntax#process-typed diff --git a/docs/process.mdx b/docs/process.mdx index ec0ffb4f4a..3b78fada8a 100644 --- a/docs/process.mdx +++ b/docs/process.mdx @@ -23,6 +23,10 @@ process hello { See [Process][syntax-process] for a full description of the process syntax. +:::note +This page describes the original process syntax. Nextflow also supports a [typed process syntax][process-typed-page] with statically typed inputs and outputs. See [Static typing][static-typing-page] for an overview. +::: + ## Script The `script` section defines, as a string expression, the script that is executed by the process. @@ -1345,8 +1349,10 @@ process hello { [process-reference-inputs]: ./reference/process#inputs [process-reference-outputs]: ./reference/process#outputs [process-template]: ./process#template +[process-typed-page]: ./process-typed [script-closure]: ./script#closures [script-tuples]: ./script#tuples +[static-typing-page]: ./static-typing [stdlib-types-path]: ./reference/stdlib-types#path [strict-syntax-page]: ./strict-syntax [string-interpolation]: ./script#string-interpolation diff --git a/docs/reference/operator-typed.mdx b/docs/reference/operator-typed.mdx index c480d5b3b6..df806f9281 100644 --- a/docs/reference/operator-typed.mdx +++ b/docs/reference/operator-typed.mdx @@ -9,6 +9,8 @@ description: Reference for the core Nextflow operators recommended for use with This page describes the core operators that are recommended for use with static typing. +See [Using operators with static typing](../tutorials/static-types-operators) for best practices when migrating existing code. + ## collect **`Channel collect() -> Value>`** diff --git a/docs/static-typing.mdx b/docs/static-typing.mdx new file mode 100644 index 0000000000..cafa980991 --- /dev/null +++ b/docs/static-typing.mdx @@ -0,0 +1,77 @@ +--- +title: Static typing +description: Overview of static typing in Nextflow, including typed processes, typed workflows, records, and how to migrate existing pipelines. +--- + +# Static typing + +**Static typing is optional. All existing code continues to work.** + +Static typing lets you model and validate the structure of your data as it flows through a pipeline. You add [standard Nextflow types][stdlib-types] to inputs and outputs at every level, from pipeline parameters to process and workflow outputs. The Nextflow language server uses those annotations to catch type-related errors as you write code. + +:::warning +Some static typing features are previews. The syntax and behavior may change in future releases. +::: + +## Static typing features + +Static typing is a set of related language features. Each one applies type annotations at a different stage of a pipeline, and you can adopt them independently or together: + +| Page | Description | +| ---- | -------------- | +| [Processes (typed)][process-typed-page] | Type annotations for process inputs and outputs, the `stage:` section, records, and tuples. | +| [Workflows (typed)][workflow-typed-page] | Type annotations across a workflow, covering [Typed parameters][typed-parameters-page] (the `params` block), [Typed outputs][typed-outputs-page] (the `output` block), and [Typed workflows][typed-workflows-page] (the `take:` and `emit:` sections). | +| [Operators (typed)][operator-typed-page] | The core operators recommended for static typing. | + +## Enabling static typing + +Static typing requires two settings: + +1. Enable the [strict parser][strict-syntax-page] by setting the `NXF_SYNTAX_PARSER` environment variable to `v2`: + + ```bash + export NXF_SYNTAX_PARSER=v2 + ``` + + The strict parser is enabled by default in Nextflow 26.04 and later. + +2. Set `nextflow.enable.types = true` in every script that uses typed processes or typed workflows: + + ```nextflow + nextflow.enable.types = true + ``` + + The `params` block and `output` block work without this flag. + +A script with `nextflow.enable.types = true` cannot also contain legacy process or workflow definitions. For more information about feature flags, see [Feature flags][feature-flags-page]. + +## Migrating existing pipelines + +Static typing is opt-in, and you can adopt it incrementally. To migrate an existing pipeline: + +1. Make sure your code is compatible with the [strict parser][strict-syntax-page] and refactor any patterns that aren't supported in typed workflows. For more information about preparing existing code, see [Preparing for static typing][preparing-static-types]. +2. Follow the [migration tutorial][migrating-static-types] to convert parameters, processes, and workflows to the typed syntax. +3. Replace legacy operators with their typed equivalents using [Using operators with static typing][migrating-static-types-operators]. + +## See also + +Static typing builds on functionality documented in other sections: + +- [Types][stdlib-types]: The type system that the annotations draw from, including records, collections, and dataflow types. +- [Strict syntax][strict-syntax-page]: The strict (`v2`) parser that static typing requires. +- [Feature flags][feature-flags-page]: Documents the `nextflow.enable.types` flag and related settings. +- [Syntax reference][syntax-page]: Gives the formal grammar for typed processes and workflows. + +[feature-flags-page]: ./reference/feature-flags +[migrating-static-types]: ./tutorials/static-types +[migrating-static-types-operators]: ./tutorials/static-types-operators +[operator-typed-page]: ./reference/operator-typed +[preparing-static-types]: ./tutorials/static-types#preparing-for-static-typing +[process-typed-page]: ./process-typed +[stdlib-types]: ./reference/stdlib-types +[strict-syntax-page]: ./strict-syntax +[syntax-page]: ./reference/syntax +[typed-outputs-page]: ./typed-outputs +[typed-parameters-page]: ./typed-parameters +[typed-workflows-page]: ./typed-workflows +[workflow-typed-page]: ./workflow-typed diff --git a/docs/strict-syntax.mdx b/docs/strict-syntax.mdx index b7e59213dd..322ceee39d 100644 --- a/docs/strict-syntax.mdx +++ b/docs/strict-syntax.mdx @@ -720,4 +720,4 @@ For Groovy code that is complicated or if it depends on third-party libraries, i [syntax-page]: ./reference/syntax [workflow-handlers]: ./notifications#workflow-handlers [workflow-params-legacy]: ./workflow#parameters -[workflow-typed-params]: ./workflow-typed#typed-parameters +[workflow-typed-params]: ./typed-parameters diff --git a/docs/tutorials/static-types-operators.mdx b/docs/tutorials/static-types-operators.mdx index db37943eb5..10b12022f0 100644 --- a/docs/tutorials/static-types-operators.mdx +++ b/docs/tutorials/static-types-operators.mdx @@ -7,12 +7,10 @@ description: Best practices for using operators with static typing in Nextflow. Nextflow 26.04 brings updates to the operator library in order to support static tying and records. This page provides best practices for using operators with static typing. -See [Migrating to static typing][migrating-static-types] for more information about migrating pipelines to static typing. - -## Overview - All operators can be used with or without static typing (i.e. [typed workflows][workflow-typed-page]). However, only a core subset of operators are recommended for use with static typing, while the rest are discouraged. They are distinguished here as *core operators* and *legacy operators*. +See [Migrating to static typing][migrating-static-types] for more information about migrating pipelines to static typing. + ## Core operators The [core operators][operator-typed-page] are recommended for use with static typing. When static typing is enabled (via `nextflow.enable.types`), some of these operators have stricter semantics which may require minor changes to pipeline code. These cases are described below. diff --git a/docs/typed-outputs.mdx b/docs/typed-outputs.mdx new file mode 100644 index 0000000000..75ba40debc --- /dev/null +++ b/docs/typed-outputs.mdx @@ -0,0 +1,74 @@ +--- +title: Typed outputs +description: Annotate workflow outputs in Nextflow using type annotations in the output block. +--- + +# Typed outputs + + + +Workflow outputs can use type annotations: + +```nextflow +nextflow.enable.types = true + +params { + input: String +} + +process fastqc { + input: + reads: Path + + output: + file('fastqc') + + // ... +} + +process summary { + input: + logs: Path + + output: + file('report.html') + + // ... +} + +workflow { + main: + ch_reads = channel.fromPath(params.input) + ch_fastqc = fastqc(ch_reads) + summary_report = summary(ch_fastqc.collect()) + + publish: + fastqc = ch_fastqc + summary_report = summary_report +} + +output { + fastqc: Channel { + path '.' + } + + summary_report: Path { + path '.' + } +} +``` + +In the above example, the workflow declares two outputs: + +- `fastqc`: a channel of FastQC results (`Channel`) +- `summary_report`: a dataflow value containing a summary report (`Value` or `Path`) + +## Features + +Type annotations document the structure of each workflow output, and the language server uses them to validate the type of each published output. + +Outputs that receive a dataflow value can be declared as `Value` or `V` for short. In this example, the `summary_report` is declared with type `Path` as a shorthand for `Value`. + +For more information about standard types, see [Types][stdlib-types]. + +[stdlib-types]: ./reference/stdlib-types diff --git a/docs/typed-parameters.mdx b/docs/typed-parameters.mdx new file mode 100644 index 0000000000..219283f8f7 --- /dev/null +++ b/docs/typed-parameters.mdx @@ -0,0 +1,45 @@ +--- +title: Typed parameters +description: Declare statically typed pipeline parameters in Nextflow using the params block. +--- + +# Typed parameters + + + +A script can declare parameters using the `params` block: + +```nextflow +params { + // Path to input data. + input: Path + + // Whether to save intermediate files. + save_intermeds: Boolean +} + +workflow { + analyze(params.input, params.save_intermeds) +} +``` + +## Features + +Parameters can use all [standard types][stdlib-types] except the dataflow types (`Channel` and `Value`). + +Typed parameters should only be referenced in the entry workflow or `output` block. Parameters can be passed to workflows and processes as explicit inputs. + +The command line, params file, or config file can override the default value. Parameters from multiple sources are resolved in the order described in [Pipeline parameters][cli-params]. Nextflow converts command-line parameters to the type set by their annotation. + +A parameter that doesn't specify a default value is a *required* parameter. If a required parameter has no value at runtime, the run fails. + +For more information about standard types, see [Types][stdlib-types]. + + + +Boolean parameters that don't specify a default value default to `false`. + + + +[cli-params]: ./cli#pipeline-parameters +[stdlib-types]: ./reference/stdlib-types diff --git a/docs/typed-workflows.mdx b/docs/typed-workflows.mdx new file mode 100644 index 0000000000..fd89acc72a --- /dev/null +++ b/docs/typed-workflows.mdx @@ -0,0 +1,59 @@ +--- +title: Typed workflows +description: Add type annotations to the take and emit sections of Nextflow workflows. +--- + +# Typed workflows + + + +:::warning +Typed workflows are a preview feature. The syntax and behavior may change in future releases. +::: + +Typed workflows can use type annotations in the `take:` and `emit:` sections: + +```nextflow +nextflow.enable.types = true + +workflow hello_bye { + take: + samples: Channel + + main: + ch_hello = hello(samples) + val_bye = bye(ch_hello.collect()) + + emit: + result: Value = val_bye +} +``` + +In the above example, `hello_bye` takes a channel of files (`Channel`) and emits a dataflow value with a single file (`Value`). For more information about the available types, see [Types][stdlib-types]. + +## Restricted syntax + +The following syntax patterns are no longer supported in typed workflows: + +- Using `Channel` to access channel factories (use `channel` instead) + +- Using implicit closure parameters (declare parameters explicitly instead) + +- Using `set` or `tap` to assign channels (use standard assignments instead) + +- Composing dataflow logic with `|` and `&` (use standard method calls instead) + +- Accessing process and workflow outputs via `.out` (use standard assignments instead) + +For more information about preparing existing code, see [Preparing for static typing][preparing-static-types]. + +## Operators + +The operator library supports static typing and records. All operators work in both typed and legacy workflows, but only a [core subset][operator-typed-page] of operators is recommended for static typing. + +For more information about best practices when migrating existing code, see [Using operators with static typing][migrating-static-types-operators]. + +[migrating-static-types-operators]: ./tutorials/static-types-operators +[operator-typed-page]: ./reference/operator-typed +[preparing-static-types]: ./tutorials/static-types#preparing-for-static-typing +[stdlib-types]: ./reference/stdlib-types diff --git a/docs/workflow-typed.mdx b/docs/workflow-typed.mdx index e51c69ba20..b9997990b9 100644 --- a/docs/workflow-typed.mdx +++ b/docs/workflow-typed.mdx @@ -5,115 +5,29 @@ description: Use typed workflows in Nextflow to define statically typed inputs a # Workflows (typed) -Typed workflows use a new syntax for inputs and outputs that supports static typing. +Typed workflows extend static typing to the workflow level. Type annotations apply to three parts of a workflow, and the Nextflow language server uses them to validate data as it enters, moves through, and leaves a pipeline. -To use this feature: +The features compose into an end-to-end chain. A value typed in the `params` block flows into a workflow's `take:` section, through its processes, and out through the `output` block. The language server checks each handoff as you write code. -1. Enable the [strict parser][strict-syntax-page] by setting the `NXF_SYNTAX_PARSER` environment variable to `v2`: - - ```bash - export NXF_SYNTAX_PARSER=v2 - ``` - -2. Set `nextflow.enable.types = true` in every script that uses typed workflows. The `params` block and `output` block can be used without this feature flag. - -See [Workflow (typed)][syntax-workflow-typed] for the complete syntax reference and [Migrating to static typing][migrating-static-types] to migrate existing code to static typing. +:::note +Typed workflows require static typing to be enabled. For more information about the required settings, see [Enabling static typing][enabling-static-typing]. +::: ## Typed parameters -A script can declare parameters using the `params` block: - -```nextflow -params { - // Path to input data. - input: Path - - // Whether to save intermediate files. - save_intermeds: Boolean -} +Type annotations on the `params` block type a pipeline's parameters. The language server validates each parameter reference against its declared type. -workflow { - analyze(params.input, params.save_intermeds) -} -``` - -All [standard types][stdlib-types] except for the dataflow types (`Channel` and `Value`) can be used for parameters. - -Typed parameters should only be referenced in the entry workflow or `output` block. Parameters can be passed to workflows and processes as explicit inputs. - -The default value can be overridden by the command line, params file, or config file. Parameters from multiple sources are resolved in the order described in [Pipeline parameters][cli-params]. Parameters specified on the command line are converted to the appropriate type based on the corresponding type annotation. - -A parameter that doesn't specify a default value is a *required* parameter. If a required parameter is not given a value at runtime, the run will fail. - - - -Boolean parameters that don't specify a default value will default to `false`. +See [Typed parameters][typed-parameters-page] for more information. ## Typed outputs -Workflow outputs can use type annotations: - -```nextflow -nextflow.enable.types = true - -params { - input: String -} - -process fastqc { - input: - reads: Path - - output: - file('fastqc') - - // ... -} - -process summary { - input: - logs: Path - - output: - file('report.html') - - // ... -} +Type annotations on the `output` block type a workflow's published results. The language server validates each published output against its declared type. -workflow { - main: - ch_reads = channel.fromPath(params.input) - ch_fastqc = fastqc(ch_reads) - summary_report = summary(ch_fastqc.collect()) - - publish: - fastqc = ch_fastqc - summary_report = summary_report -} - -output { - fastqc: Channel { - path '.' - } - - summary_report: Path { - path '.' - } -} -``` - -In the above example, the workflow declares two outputs: - -- `fastqc`: a channel of FastQC results (`Channel`) -- `summary_report`: a dataflow value containing a summary report (`Value` or `Path`) - -Type annotations are useful for documenting the structure of each workflow output, and they can be used by the language server to validate the type of each published output. - -Outputs that receive a dataflow value can be declared as `Value` or `V` for short. In this example, the `summary_report` is declared with type `Path` as a shorthand for `Value`. +See [Typed outputs][typed-outputs-page] for more information. ## Typed workflows @@ -123,53 +37,17 @@ Outputs that receive a dataflow value can be declared as `Value` or `V` for s Typed workflows are a preview feature. The syntax and behavior may change in future releases. ::: -Typed workflows can use type annotations in the `take:` and `emit:` sections: - -```nextflow -nextflow.enable.types = true - -workflow hello_bye { - take: - samples: Channel - - main: - ch_hello = hello(samples) - val_bye = bye(ch_hello.collect()) - - emit: - result: Value = val_bye -} -``` - -In the above example, `hello_bye` takes a channel of files (`Channel`) and emits a dataflow value with a single file (`Value`). See [Types][stdlib-types] for the list of available types. - -### Restricted syntax - -The following syntax patterns are no longer supported in typed workflows: - -- Using `Channel` to access channel factories (use `channel` instead) - -- Using implicit closure parameters (declare parameters explicitly instead) - -- Using `set` or `tap` to assign channels (use standard assignments instead) - -- Composing dataflow logic with `|` and `&` (use standard method calls instead) - -- Accessing process and workflow outputs via `.out` (use standard assignments instead) - -See [Preparing for static typing][preparing-static-types] for more information. +Type annotations in the `take:` and `emit:` sections type a workflow's inputs and outputs. The language server validates each input and output against its declared type. -### Operators +See [Typed workflows][typed-workflows-page] for more information. -The operator library has been updated to provide first-class support for static typing and records. All operators can be used in both typed workflows and legacy workflows. However, only a [core subset][operator-typed-page] of operators are recommended for use with static typing. +## See also -See [Using operators with static typing][migrating-static-types-operators] for best practice guidelines when migrating existing code. +For the complete syntax reference, see [Workflow (typed)][syntax-workflow-typed]. For more information about converting existing code, see [Migrating to static typing][migrating-static-types]. -[cli-params]: ./cli#pipeline-parameters +[enabling-static-typing]: ./static-typing#enabling-static-typing [migrating-static-types]: ./tutorials/static-types -[migrating-static-types-operators]: ./tutorials/static-types-operators -[operator-typed-page]: ./reference/operator-typed -[preparing-static-types]: ./tutorials/static-types#preparing-for-static-typing -[stdlib-types]: ./reference/stdlib-types -[strict-syntax-page]: ./strict-syntax [syntax-workflow-typed]: ./reference/syntax#workflow-typed +[typed-outputs-page]: ./typed-outputs +[typed-parameters-page]: ./typed-parameters +[typed-workflows-page]: ./typed-workflows diff --git a/docs/workflow.mdx b/docs/workflow.mdx index 1252a320be..a7bee388c0 100644 --- a/docs/workflow.mdx +++ b/docs/workflow.mdx @@ -13,6 +13,10 @@ In Nextflow, a **workflow** is a specialized function for composing [processes][ - Both entry workflows and named workflows can contain [dataflow logic](#dataflow) such as calling processes, workflows, and channel operators. +:::note +This page describes the original workflow syntax. Nextflow also supports a [typed workflow syntax][workflow-typed-page] with statically typed inputs and outputs. See [Static typing][static-typing-page] for an overview. +::: + ## Entry workflow A script can define up to one *entry workflow*, which does not have a name and serves as the entrypoint of the script: @@ -778,5 +782,7 @@ Workflows can also be invoked recursively: [stdlib-types-value]: ./reference/stdlib-types#valuev [syntax-include]: ./reference/syntax#include [workflow-outputs-first-preview]: ./migrations/24-04#workflow-outputs-first-preview +[static-typing-page]: ./static-typing [workflow-outputs-second-preview]: ./migrations/24-10#workflow-outputs-second-preview [workflow-outputs-third-preview]: ./migrations/25-04#workflow-outputs-third-preview +[workflow-typed-page]: ./workflow-typed