module binaries should not require Wave

[stevekm]

I am investigating the usage of Nextflow's module binaries to be able to bundle per-module bin directory and scripts, as described here https://www.nextflow.io/docs/latest/module.html#module-binaries

Notably, it seems like when using cloud execution such as AWS Batch, it is a requirement to use Wave containers;

This feature requires the use of a local or shared file system for the pipeline work directory, or Wave containers when using cloud-based executors.

This seems like an arbitrary and unnecessary requirement. Especially considering that the top-level pipeline bin directory does not have this requirement. In fact if you inspect the Nextflow .command.run file you can see how this one is handled;

nxf_main() {
....
    /home/ec2-user/miniconda/bin/aws s3 cp --recursive --only-show-errors s3://my-bucket/tmp/2d/1c2bb380285efdd60198d5406f58d0/bin $PWD/nextflow-bin
    chmod +x $PWD/nextflow-bin/* || true
    export PATH=$PWD/nextflow-bin:$PATH
.....

Its clear that Nextflow already has the capability to upload the workflow bin dir to S3, then pull it back down inside the running Nextflow job and add it to the $PATH during execution. So it does not really make sense why such a heavyweight solution like Wave would be required to do the exact same thing for the module's bin dir.

"Just use Wave containers" is not really a good solution for this considering that Wave is an external service dependency, which was not previously needed for this same functionality elsewhere in Nextflow, and usage of Wave is not feasible if your infrastructure has other considerations that are not addressed by the implementation of Wave. So the end result is, if you cannot run Wave, or if Wave is not available, you just cannot use this Nextflow feature. This does not seem like a good position to be in, as a pipeline developer.

I think a better solution for this, would be that Nextflow just uses the same behaviour for module binaries as it does for the workflow's bin dir, cache it in S3 during execution and pull it back down locally on the job's EC2 and add it as an extra location in PATH. Its not really clear to me why this is not already the default behavior. Maybe we can get this behavior enabled as an option instead of forcing Wave usage?

[pditommaso]

I understand this can be seen an unnecessary limitation, however are technical reasons why it's not supported.

First of all some context. Module binaries is an opt-in feature because it's not generally supported by all executors. Therefore we requires the users to enable it explicitly to be aware of the exiting limitations.

Currently it's supported by 1) local and grid executors and 2) cloud executors via Wave. The reason for this relies in the underlying storage used by each executor.

In the first case, since those "binary" accessible either in the local or shared file system, it's enough adding those path to the task PATH to make them available.

When using a cloud executor, things are more complicated because the lack of a shared file system would require to upload those scripts in the object storage and then download in the container somewhere accessible to the task PATH.

You would argue that is what's done for workflow level bin/ directly, then why it's not the same for module?

One reason is that in case of workflow binaries, being a single, predictable path, it's fairly simple to upload to a bucket, and then download in the tasks. Having many modules each of them with its own binaries, would required to handle different namespaces and some synchronization strategy to avoid multiple runs of the same process to upload the same binaries multiple times, etc. Not impossible, but unneeded complexity having a better alternative.

Another reason is that the ability to share the workflow bin/ directory comes from the early days of nextflow, in which uploading to the object storage and downloading them using aws command installed in the VM was the only solution viable.

However this approach, it's extremely fragile, hard to configure, error-prone, sub-optimal in term of performance and an hassle to maintain. Without counting that it requires a different ugly glue code, like the one you are reported, for each different cloud vendor.

This the reason we decided to implement the support for modules binaries for cloud executors using Wave. This approach allows us to rely on the augmentation capabilities to include the tasks binaries directly in the container, with the following advantages:

1. general solution, work with any cloud out-of-the-box;
2. no need to manage race-conditions uploading the script, transparently managed by wave;
3. no need for CLI tool to be pre-installed in the VM or container;
4. no messy command line wrangling to copy those files.

Yes, requires an extra service, however it's provided OSS and free of charge to any Platform user or it cab even be installed on-prem. I think it's a fair assumption relying on a web-service when working with cloud and containers, otherwise not even docker.io should be an option.

Last but not least, we consider the access of S3 files in nextflow batch jobs via command line (similarly for other cloud) a legacy solution for the reasons mentioned (unreliable, hard to configure and maintain, etc) and plan to converge more over the years to service-based solutions in which Wave and Fusion are central components to deploy pipelines at scale in the (any) cloud.

[stevekm]

Hi Paolo, thanks for the detailed response. I appreciate the work you guys are putting into Nextflow and its surrounding ecosystem and supporting services. The reasoning described here appears sound to me.

I guess my concern then is the situation that developers are placed in due to this reliance on external services.

Its fair to say that Nextflow already depended on external services such as DockerHub, S3, etc.. However I think there is a key difference here, in that previously the external components that Nextflow relied on were already well-established and widely used technologies. For example, the homepage of Nextflow defended its reliance on bash Linux because Linux was perceived to be "the lingua franca of computing". Similarly DockerHub, S3, and others were already widely used and adopted platforms and services independent of Nextflow and Bioinformatics. And, most all of these services were generally optional to use Nextflow's features. Nextflow has been spectacular as a "batteries included" framework that "just works" out of the box with whatever combination of infra and services you are likely already using. This has been a key driver in my continuing choice to use Nextflow over other frameworks for pipelines, because it "just works" with whatever infra you likely already have, its basically a drop-in replacement for whatever methodology you were already using.

So this is a significant change to now shift Nextflow from "works with what you are already using" to "you need to spin up these Nextflow-specific services for it to work". Wave and Fusion are both relatively new and Nextflow-specific offerings, in contrast to other services which were well established with broad adoption across industries. And this is not the first time that Nextflow features have been reliant on external services; I recall that the usage of Nextfow 'secrets' with AWS Batch execution was (is?) reliant on Seqera Platform.

One of the results of this reliance on external services for usage of Nextflow's great new features, is that developers who are unable to use those features are now forced to write their Nextflow code in worse, less ideal ways, seeking work-arounds and band-aids due to the feature they need being gate-kept behind a service they might not be able to use. It seems like organizations like nf-core and others might suffer from this particularly, due to their pipelines' tendency to have the philosophy of "run anywhere" and thus being forced to target programming methodologies that are compatible with the lowest-common-denominator instead of being able to use better Nextflow features that would improve code quality for everyone. This example of "module binaries" is one of such cases, I think. This has repercussions for the entire Nextflow community, because it will cause codebases to split into two camps; "able to use premium Nextflow features" and "not able to use premium Nextflow features". Considering the wide-spread adoption that nf-core is seeing, and how the Nextflow community's pipelines are being promoted more heavily across the industry, the results of these decisions are going to trickle-down to everyone.

The other really important consideration for this, is that Seqera's Wave is a free open source technology and service, for now. But the past decade of computing has seen plenty of examples of free services undergoing payment structure changes, or open source software undergoing licensing changes, which disrupted the communities built around them. Some notable examples include DockerHub's implementation of API limits and paid tiers, which drove many, including nf-core, to quay.io. And I believe ElasticSearch, Terraform, and Redis all saw licensing or business shifts that disrupted their open source communities. When this happens it also bifurcates the developer community into the "have's" and the "have-nots" based on which developers are supported by organizations who can adapt to the changing business requirements to implement that software. But its usually the "have-nots" who are producing the public code that the rest of the community is relying on. And, its that very same public code which is a driving factor in the choice to adopt Nextflow and its related services.

So while the situation is fine today, it seems like this puts developers in the position to potentially have the rug pulled out from under them in the future if there is some change in the business policies surrounding these external services. And this would affect everyone, even those who can afford to adapt to whatever the future holds, because of the reliance on the public code produced by the community.

This is not a rebuttal to any of the technical reasons for why these decisions were made to support Nextflow's ongoing development, but more of an observation that this shift in direction changes the context of Nextflow in a way that is a bit unexpected and quite different from what we've seen in the past.

[cariaso]

it seems like this puts developers in the position to potentially have the rug pulled out from under them in the future if there is some change in the business policies surrounding these external services.

There is no longer a community edition of Tower (now Seqera Platform). Or rather, we stopped updating the community edition.
(https://nextflow.slack.com/archives/C02T98A23U7/p1753369556052889?thread_ts=1753216952.614859&cid=C02T98A23U7)

This is a very real concern