New RFC: switching to topic channels for collecting versions

[maxulysse]

Have you read the RFC docs?

• Yes, I have read and understood the RFC docs

Summary

Topic channels are now (as of Nextflow 25.04) stable.
This would greatly simplify the way we can handle versions collection in the scripts.

Champion

@maxulysse

Background & Motivation

Mixing MODULES.out.versions has been our way to deal with collecting modules versions.
But it leads to a lot of redundant code, and it's not looking too nice.
Topic was created to relieve that problem.
It is now stable enough, so we should begin switching to it.

Year old plan from @mashehu and @mirpedrol

https://github.com/nf-core/modules/issues/4517#issuecomment-2194150864

Progressively move to topics, break dependency with nf-core/modules#5834
Use a topic name versions_file with old versions.yml path output style
Use a topic name versions for a new eval syntax
Have pipeline-level logic to load the old files and process into a string that matches the new eval syntax
Pipeline logic to take the new style strings and generate the required YAML files for reporting
This way we can progressively switch to the new eval syntax in modules, which will be a rather large and manual effort.

Same with topics - the old channels will remain for now with the same channel names, so the old syntax should continue to work.

Goals

• Use topic in modules/subworkflows
• Use topic in pipelines

Non-Goals

No response

References

• Initial issue that led to the creation of topic
• One of many PR's by Ben showcasing the possibilities
• Original Modules issue tracking it

[nf-core-bot]

RFC approval status: ✅ Approved

RFC has approvals from 9/7 required @core-team quorum.

Review Status	Core Team members
✅ Approved	@ewels, @FriederikeHanssen, @kenibrewer, @edmundmiller, @jfy133, @mribeirodantas, @nvnieuwk, @sateeshperi, @JoseEspinosa
🕐 Pending	@maxulysse, @mashehu, @mirpedrol, @FranBonath, @christopher-hakkaart

[maxulysse]

I co-champion @edmundmiller and @FriederikeHanssen

[ewels]

Sounds good to me, please add a bit more detail in the next step of the RFC (the PR) showing a couple of example code snippets about what should be added and what should be removed, basically as a sort of guide for people.

It'd also be nice to mention the downsides in RFC proposals. In this case, it's going to be a lot of refactoring for people. It should be possible to do it progressively though right? This kind of discussion and thinking about what work needs to be done and how is ideal for RFCs 😊

/approve

[mahesh-panchal]

This will allow easier use of storeDir too.

One thing though is should the command be in an eval or should we use an environment variable generated in the script and test it matches the version number pattern or hashcode pattern? There are currently no checks that the emitted value is what it's supposed to be and sometimes causes problems downstream.

[FriederikeHanssen]

/approve

On Phils comment, I think at some point we wanted to test if we can mix and match topics channels with the regular old version channel. I forgot the outcome of that. If we can that should lessen the burden on pipeline developers and prevent us from being in a weird locked state where we either need to have both or do the entire update in a single monster modules Pr.

[maxulysse]

I do think mixing both work, I'm fairly sure we tested that already.
But yeah, it would be nice to be sure.
I'll make some POC for this in the PR, and plan the full migration.

[kenibrewer]

/approve

Also, I agree with Phil that ideally, these RFC proposals would be fleshed out with a bit more detail about the pros and cons. Would you mind adding some more context @maxulysse ?

[edmundmiller]

/approve

Updated some info from past issues.

I think we start this piece-meal, like the nf-test migration, and see if anything breaks, then we can try to automate it. I think prior to this, we migrate to nf-utils, and the logic to handle different styles can be housed there.

[jfy133]

/approve