Skip to content

Commit 583285c

Browse files
authored
Module System Implementation (#6768)
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
1 parent f145979 commit 583285c

File tree

60 files changed

+9187
-411
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

60 files changed

+9187
-411
lines changed

docs/cli.md

Lines changed: 130 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -262,7 +262,135 @@ $ nextflow secrets set AWS_ACCESS_KEY_ID
262262
$ nextflow secrets delete AWS_ACCESS_KEY_ID
263263
```
264264

265-
See {ref}`cli-secrets` for more information.
265+
See {ref}`cli-secrets` for more information.
266+
267+
## Module management
268+
269+
:::{versionadded} 26.04.0
270+
:::
271+
272+
Module management commands enable working with reusable, registry-based modules. The Nextflow module system allows you to install, run, search, and publish standardized modules from registries, eliminating duplicate work and spreading improvements throughout the community.
273+
274+
Use these commands to discover modules in registries, install them into your project, run them directly without creating a workflow, and publish your own modules for others to use.
275+
276+
### Installing modules
277+
278+
The `module install` command downloads modules from a registry and makes them available in your workflow. Modules are stored locally in the `modules/` directory. An additional `.module-info` file is created during to store installation information such as the module checksum at installation and the registry URL.
279+
280+
Use this to add reusable modules to your pipeline, manage module versions, or update modules to newer versions.
281+
282+
```console
283+
$ nextflow module install nf-core/fastqc
284+
$ nextflow module install nf-core/fastqc -version 1.0.0
285+
```
286+
287+
After installation, module will be available in `modules/nf-core/fastqc`.
288+
289+
Use the `-force` flag to reinstall a module even if local modifications exist.
290+
291+
See {ref}`cli-module-install` for more information.
292+
293+
### Running modules directly
294+
295+
The `module run` command executes a module directly from the registry without requiring a wrapper workflow. This provides immediate access to module functionality for ad-hoc tasks or testing.
296+
297+
Use this to quickly run a module, test module functionality, or execute one-off data processing tasks.
298+
299+
```console
300+
$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz'
301+
$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0
302+
```
303+
304+
The command accepts all standard Nextflow execution options (`-profile`, `-resume`, etc.):
305+
306+
```console
307+
$ nextflow module run nf-core/salmon \
308+
--reads reads.fq \
309+
--index salmon_index \
310+
-profile docker \
311+
-resume
312+
```
313+
314+
See {ref}`cli-module-run` for more information.
315+
316+
### Listing modules
317+
318+
The `module list` command displays all modules currently installed in your project, showing their versions and integrity status.
319+
320+
Use this to review installed modules, check module versions, or detect local modifications.
321+
322+
```console
323+
$ nextflow module list
324+
$ nextflow module list -output json
325+
```
326+
327+
The output shows each module's name, installed version, and whether it has been modified locally. Use `-json` for machine-readable output suitable for scripting.
328+
329+
See {ref}`cli-module-list` for more information.
330+
331+
### Searching for modules
332+
333+
The `module search` command queries the module registry to discover available modules by keyword or name.
334+
335+
Use this to find modules for specific tasks, explore available tools, or discover community contributions.
336+
337+
```console
338+
$ nextflow module search alignment
339+
$ nextflow module search "quality control" -limit 10
340+
$ nextflow module search bwa -output json
341+
```
342+
343+
Results include module names, versions, descriptions, and download statistics. Use `-limit` to control the number of results and `-output json` for programmatic access.
344+
345+
See {ref}`cli-module-search` for more information.
346+
347+
### Viewing module information
348+
349+
The `module info` command displays detailed metadata and usage information for a specific module from the registry.
350+
351+
Use this to understand module requirements, view input/output specifications, see available tools, or generate usage templates before installing or running a module.
352+
353+
```console
354+
$ nextflow module info nf-core/fastqc
355+
$ nextflow module info nf-core/fastqc -version 1.0.0
356+
$ nextflow module info nf-core/fastqc -output json
357+
```
358+
359+
The output includes the module's version, description, authors, keywords, tools, input/output channels, and a generated usage template showing how to run the module. Use `-json` for machine-readable output suitable for programmatic access.
360+
361+
See {ref}`cli-module-info` for more information.
362+
363+
### Removing modules
364+
365+
The `module remove` command deletes modules from your project, removing local files and configuration entries.
366+
367+
Use this to clean up unused modules, free disk space, or remove deprecated modules from your pipeline.
368+
369+
```console
370+
$ nextflow module remove nf-core/fastqc
371+
$ nextflow module remove nf-core/fastqc -keep-files
372+
```
373+
374+
By default, both local files and configuration entries are removed. Use `-keep-files` to remove the configuration entry and `.module-info` while keeping local files.
375+
376+
See {ref}`cli-module-remove` for more information.
377+
378+
### Publishing modules
379+
380+
The `module publish` command uploads modules to a registry, making them available for others to install and use.
381+
382+
Use this to share your modules with the community, contribute to module libraries, or distribute modules within your organization.
383+
384+
```console
385+
$ nextflow module publish myorg/my-module
386+
$ nextflow module publish myorg/my-module -dry-run
387+
```
388+
389+
Publishing requires authentication via the `NXF_REGISTRY_TOKEN` environment variable or `registry.apiKey` in the Nextflow configuration. The module must include `main.nf`, `meta.yaml`, and `README.md` files.
390+
391+
Use `-dry-run` to validate your module structure without uploading.
392+
393+
See {ref}`cli-module-publish` for more information.
266394

267395
## Configuration and validation
268396

@@ -384,7 +512,7 @@ Use this to understand input/output relationships between tasks, trace data flow
384512
$ nextflow lineage
385513
```
386514

387-
See {ref}`data-lineage-page` to get started and {ref}`cli-lineage` for more information.
515+
See {ref}`data-lineage-page` to get started and {ref}`cli-lineage` for more information.
388516

389517
## Seqera Platform
390518

docs/module.md

Lines changed: 158 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -279,8 +279,165 @@ This feature requires the use of a local or shared file system for the pipeline
279279

280280
## Sharing modules
281281

282-
Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. While Nextflow does not provide an explicit mechanism for sharing modules, there are several ways to do it:
282+
Modules are designed to be easy to share and re-use across different pipelines, which helps eliminate duplicate work and spread improvements throughout the community. There are several ways to share modules:
283283

284+
- Use the Nextflow module registry (recommended, see below)
284285
- Simply copy the module files into your pipeline repository
285286
- Use [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to fetch modules from other Git repositories without maintaining a separate copy
286287
- Use the [nf-core](https://nf-co.re/tools#modules) CLI to install and update modules with a standard approach used by the nf-core community
288+
289+
(module-registry)=
290+
291+
## Registry-based modules
292+
293+
:::{versionadded} 26.04.0
294+
:::
295+
296+
Nextflow provides a module registry that enables you to install, share, and manage modules from centralized registries. This system provides version management, integrity checking, and seamless integration with the Nextflow DSL.
297+
298+
### Installing modules from a registry
299+
300+
Use the `module install` command to download modules from a registry:
301+
302+
```console
303+
$ nextflow module install nf-core/fastqc
304+
$ nextflow module install nf-core/fastqc -version 1.0.0
305+
```
306+
307+
Installed modules are stored in the `modules/` directory and can be included by name instead of a relative path:
308+
309+
```nextflow
310+
include { FASTQC } from 'nf-core/fastqc'
311+
312+
workflow {
313+
reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz')
314+
FASTQC(reads)
315+
}
316+
```
317+
318+
### Running modules directly
319+
320+
For ad-hoc tasks or testing, you can run a module directly without creating a workflow:
321+
322+
```console
323+
$ nextflow module run nf-core/fastqc --input 'data/*.fastq.gz'
324+
```
325+
326+
This command accepts all standard Nextflow options (`-profile`, `-resume`, etc.) and automatically downloads the module if not already installed.
327+
328+
329+
### Discovering modules
330+
331+
Search for available modules using the `module search` command:
332+
333+
```console
334+
$ nextflow module search alignment
335+
$ nextflow module search "quality control" -limit 10
336+
```
337+
338+
List installed modules in your project:
339+
340+
```console
341+
$ nextflow module list
342+
```
343+
344+
### Module checksum verification
345+
346+
Nextflow automatically verifies module integrity using checksums. If you modify a module locally, Nextflow will detect the change and prevent accidental overwrites:
347+
348+
```console
349+
$ nextflow module install nf-core/fastqc -version 1.1.0
350+
Warning: Module nf-core/fastqc has local modifications. Use -force to override.
351+
```
352+
353+
Use the `-force` flag to override local modifications when needed.
354+
355+
### Removing modules
356+
357+
Use the `module remove` command to uninstall a module:
358+
359+
```console
360+
$ nextflow module remove nf-core/fastqc
361+
```
362+
363+
By default, both the module files and the `.module-info` file are removed. Use the flags below to control this behaviour:
364+
365+
- `-keep-files` — Remove the `.module-info` file created at install but keep the rest of files
366+
- `-force` — Force removal even if the module has no `.module-info` file (i.e. not installed from a registry) or has local modifications
367+
368+
### Viewing module information
369+
370+
Use the `module info` command to display metadata and a usage template for a module:
371+
372+
```console
373+
$ nextflow module info nf-core/fastqc
374+
$ nextflow module info nf-core/fastqc -version 1.0.0
375+
```
376+
377+
The output includes the module description, authors, keywords, tools, inputs, outputs, and a ready-to-use command-line template. Use `-json` to get machine-readable output.
378+
379+
### Publishing modules
380+
381+
To share your own modules, use the `module publish` command:
382+
383+
```console
384+
$ nextflow module publish myorg/my-module
385+
```
386+
387+
The argument can be either a `scope/name` reference (for an already-installed module) or a local directory path containing the module files.
388+
389+
Your module directory must include:
390+
391+
- `main.nf`: The module entry point
392+
- `meta.yaml`: Module metadata (name, description, version, etc.)
393+
- `README.md`: Module documentation
394+
395+
Authentication is required for publishing and can be provided via the `NXF_REGISTRY_TOKEN` environment variable or in your configuration:
396+
397+
```groovy
398+
registry {
399+
apiKey = 'YOUR_REGISTRY_TOKEN'
400+
}
401+
```
402+
403+
Use `-dry-run` to validate your module structure without uploading:
404+
405+
```console
406+
$ nextflow module publish myorg/my-module -dry-run
407+
```
408+
409+
### Registry configuration
410+
411+
By default, Nextflow uses the public registry at `https://registry.nextflow.io`. You can configure alternative or additional registries:
412+
413+
```groovy
414+
registry {
415+
url = [
416+
'https://private.registry.myorg.com',
417+
'https://registry.nextflow.io'
418+
]
419+
apiKey = '${MYORG_TOKEN}'
420+
}
421+
```
422+
423+
Registries are queried in the order specified until a module is found. The `apiKey` is used only for the primary (first) registry.
424+
425+
### Module directory structure
426+
427+
Registry modules follow a standard directory structure:
428+
429+
```
430+
modules/
431+
└── scope/
432+
└── module-name/
433+
├── .checksum # Integrity checksum (generated automatically)
434+
├── README.md # Documentation (required for publishing)
435+
├── main.nf # Module entry point (required)
436+
├── meta.yaml # Module metadata (required for publishing)
437+
├── resources/ # Optional: module binaries and resources
438+
└── templates/ # Optional: process templates
439+
```
440+
441+
The `modules/` directory should be committed to your Git repository to ensure reproducibility.
442+
443+
See the {ref}`cli-page` documentation for more information about module commands.

0 commit comments

Comments
 (0)