Modules: ideas about tagging and grouping modules

[mirpedrol]

Hello!

As part of a project I am working on in the lab, I am working on a way to group modules into what we call "classes". These are modules that have the same input and output and perform the same operation. These modules are interchangeable between them.
This is very useful for us since we are interested in benchmarking different tools, and this would facilitate the selection of the best tool. Additionally, it can also be useful to tag the modules to allow LLMs to easily select them when constructing workflows.

The tagging of input and output files in nf-core modules has been very useful so far, and I would also like to tag the operation of the module (left a comment in the spec).

What I am working on:

• Different ways to group modules:
  • Manually
  • With Seqera MCP
  • Using the embedding of modules
• Automatically tagging modules with ontologies using MCPs:
  • Development of the edamMCP
• Validation of the ontology terms:
  • For this, Nextflow could help by providing a way of running a single module using test data nextflow module run
• Benchmarking of different tools using benchmarking pipelines such as nf-core/multiplesequencealign

Other links for more information:

• using class modules in nf-core/multiplesequencealign
• the nf-class python package
• an example module repo where classes are working

This issue is to track the ideas so that we can sync on our objectives. Please feel free to comment any suggestions/concerns or ask any questions you may have :)

[pditommaso]

This a very interesting idea. From my point of view this should defined as a new "type" of module that nextflow can handle natively.

A "class" module (likely we need a better naming) would declare only the inputs and outputs contracts, without requiring a concrete nextflow script.

Implementing module should reference this class and nextflow will take of validating that matching input/output spec.

Likely in my pipeline I should be able to use a "class" module in the pipeline script like any other module, having a directive that allows resolving the target (concrete) module at runtime, via an input or a parameter.

As such I see this more as nextflow feature than a schema property.

[mirpedrol]

This would be perfect!
Actually, the initial idea was to have a way for Nextflow to understand what a class is and import it as a regular module, while allowing all possible modules to be used there.
Right now, what we are doing is including the modules in a subworkflow, which can be used in the pipeline. This works very well, but obviously requires lots of additional steps updating the subworkflow, the pipeline, etc., every time you want to add a new module to the class

[pditommaso]

Awesome, I wrote a tech document how this could work as an extension of nextflow module system, introducing the concept of Module Interface. Let's continue the the discussion in this PR.