Module System Implementation

[jorgee]

This pull request introduces a new module command-line interface for managing Nextflow modules, including installing, listing, and publishing modules. It adds a new module command with several subcommands, updates the CLI infrastructure to support unknown options for this command, and brings in new dependencies required for module packaging and extraction.

Major new features and CLI enhancements:

• Added a new CmdModule class implementing the nextflow module command, with subcommands for install, list, publish, run, remove, and search. This includes full CLI argument parsing, usage/help handling, and subcommand delegation.
• ModuleRun extends CmdRun installing the module and calling CmdRun with the downloaded module.

Launcher and CLI infrastructure updates:

• Registered the new CmdModule command in the main Launcher class so it is available as a top-level command.
• Modified the CLI parser in Launcher to allow unknown options for the module command and to pass these unknown options to the command implementation. [1] [2]
• Updated the base CmdBase class to support storing and retrieving unknown CLI options, enabling more flexible argument handling for subcommands.

Dependency management:

• Added org.apache.commons:commons-compress as a dependency for handling tar.gz extraction, which is necessary for module packaging and publishing.

Building and Testing
Current CI is failing becuase it requires a local development registry instance with modules API implemented. You will need it to update build.gradle to updated npr-api package with the new module API classes. Either with includeBuild or a Maven local repository. You will also need to deploy the registry locally and configure the registry field in nextflow.config to test the new CLI commans

Open Issue and TODOs:

• Module version temporary in nextflow_spec.json for simplicity.
• Print module run outputs
• Remote module inclusion
• Add documentation
• Extend unit and integration tests

[jorgee]

Some updates:

1.- I have added the print of process outputs when finishing the process execution.

2.- I have investigated the remote module inclusion and a possible implemetation:
When using the syntax parser V1, it is easy to implement the module resolution at IncludeDef. We can check if the module is installed and install if not exists.

However, when using syntax parser V2, the problem is that visitors and resolvers used for parsing are at nf-lang. So, even if I decouple the classes to download modules and import at nf-lang, there is no way to access the session, config builder, and we will add the npr-api classes. Not sure if it would be a good solution

So, what I have done is the following:

• In the include resolver and visitor, I just convert the @scope/module to the expected path and check if it exists. If there is no local copy of the module, it fails at compile time and suggest the user to install it. If there is a local installation, it checks the versions and integrity at IncludeDef.
  I could also check the installation of the declared modules at CmdRun, then it will only fail if a module is used and not declared.

[bentsherman]

@jorgee let's move the remote module inclusion piece out of this PR. you can put it in a separate PR if you want, either way Paolo will experiment with it separately

And for the module version config, in the interest of keeping the POC simple so that we can iterate quicker, let's keep the module versions in the JSON file. Paolo can also experiment with this separately if he wants

[jorgee]

Updated the branch as @bentsherman suggested

• Modules version temporary in nextflow_spec.json
• Remote inclusion PoC in Remote module inclusion #6815
• Added docs and tests
• It will be ready once changes in registry are aplied and new npr-api version generated

[pditommaso]

I was giving a try to this, but cannot compile it

> Task :nextflow:compileGroovy
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
startup failed:
/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 24: unable to resolve class io.seqera.npr.api.schema.v1.ModuleRelease
 @ line 24, column 1.
   import io.seqera.npr.api.schema.v1.ModuleRelease
   ^

/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 25: unable to resolve class io.seqera.npr.api.schema.v1.PublishModuleResponse
 @ line 25, column 1.
   import io.seqera.npr.api.schema.v1.PublishModuleResponse
   ^

/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 205: unable to resolve class io.seqera.npr.api.schema.v1.ModuleRelease
 @ line 205, column 37.
                return new GsonEncoder<ModuleRelease>() {}.decode(body)
                                       ^

/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 472: unable to resolve class io.seqera.npr.api.schema.v1.PublishModuleResponse
 @ line 472, column 36.
               return new GsonEncoder<PublishModuleResponse>() {}.decode(body)
                                      ^

/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 205: unable to resolve class io.seqera.npr.api.schema.v1.ModuleRelease
 @ line 205, column 37.
                return new GsonEncoder<ModuleRelease>() {}.decode(body)
                                       ^

/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 205: unable to resolve class io.seqera.npr.api.schema.v1.ModuleRelease
 @ line 205, column 37.
                return new GsonEncoder<ModuleRelease>() {}.decode(body)
                                       ^

/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 472: unable to resolve class io.seqera.npr.api.schema.v1.PublishModuleResponse
 @ line 472, column 36.
               return new GsonEncoder<PublishModuleResponse>() {}.decode(body)
                                      ^

/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 472: unable to resolve class io.seqera.npr.api.schema.v1.PublishModuleResponse
 @ line 472, column 36.
               return new GsonEncoder<PublishModuleResponse>() {}.decode(body)
                                      ^

8 errors

[jorgee]

To test this branch you need to do the following:

1. Clone the registry repo and clone branch 251215-module-binary-registry-fixes

2. Build and publish the npr-api version 0.20.1 in Maven local

$ make build
$ ./gradlew :npr-api:publishToMavenLocal 

3. Deploy a local docker registry to store modules
   $ docker run -d --name docker-registry -p 5010:5000 registry:latest

4. Run the Nextflow registry
   $ bash run.sh

5. Clone Nextflow repo and checkout this branch

6. Include the Maven local in the build.gradle file

         mavenCentral()
         maven { url 'https://repo.eclipse.org/content/groups/releases' }
         maven { url 'https://oss.sonatype.org/content/repositories/snapshots' }
+        mavenLocal()
         maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/releases" }
         maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/snapshots" }

8. Build Nextflow as usual

[jorgee]

Update:

• Add module info command
• Update to registry API to v1
• /api path expected in registry config URL
• search only prints name and description (full for -json)

To test, you must run with registry branch fix-module-release-metadata

[jorgee]

Updated docs and other fixes. To test in dev, add the following lines in nextflow.config

registry {
     url='https://registry-dev.nextflow.io/api'
     apiKey='YOUR_REGISTRY_TOKEN'
}

[pditommaso]

Module System Code Review

CRITICAL (Must Fix)

1. Logic Bug — CmdModuleSearch validation (dead code)

CmdModuleSearch.groovy:64 — if (!args && args.size() != 1) is impossible: if !args is true, && short-circuits. Should be ||.

2. Inverted Condition — CmdModuleInfo (wrong log + potential NPE)

CmdModuleInfo.groovy:102-109 — Logs "No metadata found" when metadata IS present. Additionally, if fetchRelease throws, release stays null and the code falls through to printFormattedInfo which will NPE.

3. e.printStackTrace() in production code

ModuleRegistryClient.groovy:138 — Writes directly to stderr, bypassing the logging framework. Replace with log.debug or log.error.

---

IMPORTANT (Should Fix)

5. Checksum header name mismatch between code and tests

Production code checks X-NF-Module-Checksum, but tests stub X-Checksum. Tests pass but don't actually validate the real header.

6. Files.list() / Files.walk() stream resource leaks

ModuleStorage.groovy:126,171,409, ModuleChecksum.groovy:54, CmdModulePublish.groovy:222 — These streams must be closed (try-with-resources or .withCloseable{}). Compare with ModuleStorage.groovy:345 where this is done correctly.

7. removeModuleEntry always returns true

PipelineSpec.groovy:82-96 — Doesn't check if the key actually existed before returning true. Map.remove() returns the previous value (or null); the return value should be checked.

8. Bundle sent as raw bytes in JSON

CmdModulePublish.groovy:129-135 — byte[] in Gson JSON will be huge/inefficient. Consider multipart upload.

9. Files.readAllBytes() for checksum computation

ModuleChecksum.groovy:67,131 — Loads entire files into memory. Use DigestInputStream or read in chunks for large files.

10. Missing ModulesConfig in extensions.idx

RegistryConfig is registered but ModulesConfig is not — may not be discovered by the config scope mechanism.

---

SUGGESTIONS (Nice to Have)

• Documentation code example not working #11 Typo: "diretory" → "directory" in CmdModulePublish.groovy:253
• Error resuming process with output value and command defined on multiple lines #12 Unicode checkmark in CLI output may not render on all terminals
• Show a more detailed error message when hitting GitHub rate limits #13 CmdModuleRun extends CmdRun exposes all CmdRun params — consider composition
• Very minor typo fixes #14 Duplicate config-building boilerplate across all CLI commands — extract to shared method

---

Strengths

• Zip slip / tar slip protection is correctly implemented
• Good checksum verification with cleanup on mismatch
• Solid test coverage with WireMock integration tests
• Clean CLI subcommand pattern consistent with existing CmdPlugin
• Proper Apache 2.0 headers, @CompileStatic, @Slf4j, Groovy idioms throughout

[pditommaso]

Apologies the spam, but it looks valid points ☝️

[jorgee]

Update:

• Addressed critical and important fixes except 8 (not required because bundle is limited to 1MB not compressed) and 10 (depends on where we decide to write module versions)
• Change -json by -output
• Add module URL in info command

TODO/to be decided:

• remove flags
• remove '@' (I will create another PR as it require changes also in remote inclusion)
• module versions in nextflow_spec.json or nextflow.config

[jorgee]

I have removed -keep-config for module remove. I have also changed its behaviour:

• It fails when folder exists and .module-info doesn't exist or when there are modifications
• Added the -force flag to remove in above cases
• When removing with -keep-files, it removes the entry in spec and the .module-info file

[christopher-hakkaart]

Minor suggestions

[bentsherman]

Merging into the main PR since we've resolved all of the requirements, let's continue to test and refine there

[ewels]

We don't really say in the new docs where --reads and --index come from. It'd be good to have a bit more guidance about how people should know what CLI flags to use.

Could we get nextflow module run [name] --help to list available flags? That'd be really useful!

[bentsherman]

Let's move this to #6650