Module System Implementation

[jorgee]

This pull request introduces a new module command-line interface for managing Nextflow modules, including installing, listing, and publishing modules. It adds a new module command with several subcommands, updates the CLI infrastructure to support unknown options for this command, and brings in new dependencies required for module packaging and extraction.

Major new features and CLI enhancements:

• Added a new CmdModule class implementing the nextflow module command, with subcommands for install, list, publish, run, remove, and search. This includes full CLI argument parsing, usage/help handling, and subcommand delegation.
• ModuleRun extends CmdRun installing the module and calling CmdRun with the downloaded module.

Launcher and CLI infrastructure updates:

• Registered the new CmdModule command in the main Launcher class so it is available as a top-level command.
• Modified the CLI parser in Launcher to allow unknown options for the module command and to pass these unknown options to the command implementation. [1] [2]
• Updated the base CmdBase class to support storing and retrieving unknown CLI options, enabling more flexible argument handling for subcommands.

Dependency management:

• Added org.apache.commons:commons-compress as a dependency for handling tar.gz extraction, which is necessary for module packaging and publishing.

Building and Testing
Current CI is failing becuase it requires a local development registry instance with modules API implemented. You will need it to update build.gradle to updated npr-api package with the new module API classes. Either with includeBuild or a Maven local repository. You will also need to deploy the registry locally and configure the registry field in nextflow.config to test the new CLI commans

Open Issue and TODOs:

• Module version temporary in nextflow_spec.json for simplicity.
• Print module run outputs
• Remote module inclusion
• Add documentation
• Extend unit and integration tests

[jorgee]

Some updates:

1.- I have added the print of process outputs when finishing the process execution.

2.- I have investigated the remote module inclusion and a possible implemetation:
When using the syntax parser V1, it is easy to implement the module resolution at IncludeDef. We can check if the module is installed and install if not exists.

However, when using syntax parser V2, the problem is that visitors and resolvers used for parsing are at nf-lang. So, even if I decouple the classes to download modules and import at nf-lang, there is no way to access the session, config builder, and we will add the npr-api classes. Not sure if it would be a good solution

So, what I have done is the following:

• In the include resolver and visitor, I just convert the @scope/module to the expected path and check if it exists. If there is no local copy of the module, it fails at compile time and suggest the user to install it. If there is a local installation, it checks the versions and integrity at IncludeDef.
  I could also check the installation of the declared modules at CmdRun, then it will only fail if a module is used and not declared.

[bentsherman]

@jorgee let's move the remote module inclusion piece out of this PR. you can put it in a separate PR if you want, either way Paolo will experiment with it separately

And for the module version config, in the interest of keeping the POC simple so that we can iterate quicker, let's keep the module versions in the JSON file. Paolo can also experiment with this separately if he wants

[jorgee]

Updated the branch as @bentsherman suggested

• Modules version temporary in nextflow_spec.json
• Remote inclusion PoC in POC: Remote module inclusion #6815
• Added docs and tests
• It will be ready once changes in registry are aplied and new npr-api version generated

[pditommaso]

I was giving a try to this, but cannot compile it

> Task :nextflow:compileGroovy
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
startup failed:
/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 24: unable to resolve class io.seqera.npr.api.schema.v1.ModuleRelease
 @ line 24, column 1.
   import io.seqera.npr.api.schema.v1.ModuleRelease
   ^

/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 25: unable to resolve class io.seqera.npr.api.schema.v1.PublishModuleResponse
 @ line 25, column 1.
   import io.seqera.npr.api.schema.v1.PublishModuleResponse
   ^

/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 205: unable to resolve class io.seqera.npr.api.schema.v1.ModuleRelease
 @ line 205, column 37.
                return new GsonEncoder<ModuleRelease>() {}.decode(body)
                                       ^

/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 472: unable to resolve class io.seqera.npr.api.schema.v1.PublishModuleResponse
 @ line 472, column 36.
               return new GsonEncoder<PublishModuleResponse>() {}.decode(body)
                                      ^

/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 205: unable to resolve class io.seqera.npr.api.schema.v1.ModuleRelease
 @ line 205, column 37.
                return new GsonEncoder<ModuleRelease>() {}.decode(body)
                                       ^

/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 205: unable to resolve class io.seqera.npr.api.schema.v1.ModuleRelease
 @ line 205, column 37.
                return new GsonEncoder<ModuleRelease>() {}.decode(body)
                                       ^

/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 472: unable to resolve class io.seqera.npr.api.schema.v1.PublishModuleResponse
 @ line 472, column 36.
               return new GsonEncoder<PublishModuleResponse>() {}.decode(body)
                                      ^

/Users/pditommaso/Projects/nextflow/modules/nextflow/src/main/groovy/nextflow/module/ModuleRegistryClient.groovy: 472: unable to resolve class io.seqera.npr.api.schema.v1.PublishModuleResponse
 @ line 472, column 36.
               return new GsonEncoder<PublishModuleResponse>() {}.decode(body)
                                      ^

8 errors

[jorgee]

To test this branch you need to do the following:

1. Clone the registry repo and clone branch 251215-module-binary-registry-fixes

2. Build and publish the npr-api version 0.20.1 in Maven local

$ make build
$ ./gradlew :npr-api:publishToMavenLocal 

3. Deploy a local docker registry to store modules
   $ docker run -d --name docker-registry -p 5010:5000 registry:latest

4. Run the Nextflow registry
   $ bash run.sh

5. Clone Nextflow repo and checkout this branch

6. Include the Maven local in the build.gradle file

         mavenCentral()
         maven { url 'https://repo.eclipse.org/content/groups/releases' }
         maven { url 'https://oss.sonatype.org/content/repositories/snapshots' }
+        mavenLocal()
         maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/releases" }
         maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/snapshots" }

8. Build Nextflow as usual

[pditommaso]

@jorgee i've made a force push by mistake, please check I've not removed anything

[jorgee]

Update:

• Add module info command
• Update to registry API to v1
• /api path expected in registry config URL
• search only prints name and description (full for -json)

To test, you must run with registry branch fix-module-release-metadata