Providing plugins like nf-amazon offline

[brandoncazander]

I ran into an issue recently where a pipeline failed spuriously on startup with the following error around updating the nf-amazon plugin:

Feb-11 06:26:23.294 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; plugins-dir=/root/.nextflow/plugins
Feb-11 06:26:23.295 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[[email protected]]
Feb-11 06:26:23.316 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins local root: .nextflow/plr/a9d278cd192015538bf4de338a072e51
Feb-11 06:26:23.325 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Feb-11 06:26:23.326 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Feb-11 06:26:23.330 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Feb-11 06:26:23.341 [main] ERROR org.pf4j.AbstractPluginManager - Cannot find the manifest path
org.pf4j.PluginRuntimeException: Cannot find the manifest path
    at org.pf4j.ManifestPluginDescriptorFinder.readManifest(ManifestPluginDescriptorFinder.java:75)
    at org.pf4j.ManifestPluginDescriptorFinder.find(ManifestPluginDescriptorFinder.java:56)
    at org.pf4j.AbstractPluginManager.loadPluginFromPath(AbstractPluginManager.java:812)
    at org.pf4j.DefaultPluginManager.loadPluginFromPath(DefaultPluginManager.java:130)
    at nextflow.plugin.CustomPluginManager.loadPluginFromPath(CustomPluginManager.groovy:48)
    at nextflow.plugin.LocalPluginManager.loadPluginFromPath(LocalPluginManager.groovy:81)
    at org.pf4j.AbstractPluginManager.loadPlugins(AbstractPluginManager.java:224)
    at nextflow.plugin.PluginsFacade.setup(PluginsFacade.groovy:178)
    at nextflow.plugin.Plugins.setup(Plugins.groovy:39)
    at nextflow.cli.CmdRun.run(CmdRun.groovy:276)
    at nextflow.cli.Launcher.run(Launcher.groovy:476)
    at nextflow.cli.Launcher.main(Launcher.groovy:635)
Feb-11 06:26:23.342 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-amazon version: 1.2.0
Feb-11 06:26:23.345 [main] INFO  nextflow.plugin.PluginUpdater - Downloading plugin [email protected]
Feb-11 06:26:24.384 [main] INFO  org.pf4j.util.FileUtils - Expanded plugin zip 'nf-amazon-1.2.0.zip' in 'nf-amazon-1.2.0'
Feb-11 06:26:24.387 [main] DEBUG nextflow.plugin.LocalPluginManager - Deleting existing local plugins root link: .nextflow/plr/a9d278cd192015538bf4de338a072e51/nf-amazon-1.2.0
Feb-11 06:26:24.388 [main] ERROR nextflow.cli.Launcher - .nextflow/plr/a9d278cd192015538bf4de338a072e51/nf-amazon-1.2.0
java.nio.file.DirectoryNotEmptyException: .nextflow/plr/a9d278cd192015538bf4de338a072e51/nf-amazon-1.2.0
    at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:247)
    at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
    at java.base/java.nio.file.Files.delete(Files.java:1142)
    at nextflow.plugin.LocalPluginManager.createLink0(LocalPluginManager.groovy:106)
    at nextflow.plugin.LocalPluginManager.createLinkFromPath(LocalPluginManager.groovy:94)
    at nextflow.plugin.LocalPluginManager.loadPluginFromPath(LocalPluginManager.groovy:80)
    at nextflow.plugin.PluginUpdater.load0(PluginUpdater.groovy:246)
    at nextflow.plugin.PluginUpdater.installPlugin(PluginUpdater.groovy:141)
    at nextflow.plugin.PluginUpdater.prepareAndStart(PluginUpdater.groovy:88)
    at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:289)
    at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:299)
    at nextflow.plugin.PluginsFacade.setup(PluginsFacade.groovy:179)
    at nextflow.plugin.Plugins.setup(Plugins.groovy:39)
    at nextflow.cli.CmdRun.run(CmdRun.groovy:276)
    at nextflow.cli.Launcher.run(Launcher.groovy:476)
    at nextflow.cli.Launcher.main(Launcher.groovy:635)

This the first and only time I have encountered this issue, and it was worked around by removing the .nextflow/plr/ directory—this happened before I was able to see the contents. This is using nextflow 21.08-0-edge.

My question is, can we bundle plugins like this into our compute environment and prevent nextflow from trying to download and update them at runtime? I found there is an -offline flag, but that also prevents nextflow from running a workflow from a github repository.

[abhi18av]

Hi @brandoncazander ,

Have you tried using the nextflow-VERSION-all builds? I think it might address the issues you're facing.

https://github.com/nextflow-io/nextflow/releases/tag/v21.10.6

[pditommaso]

we need to under the reason tho

[brandoncazander]

@abhi18av thanks for the response, this does indeed look like what I want. However, it's still downloading the nf-amazon plugin at runtime.

For the record, I'm trying to package a specific version of nextflow (and its plugins) in a docker image. I think this is a reproducible test case:

Image definition:

FROM amazoncorretto:17.0.1
RUN yum install -y procps-ng

FROM amazoncorretto:17.0.1
COPY --from=0 /bin/ps /bin/ps

ARG NEXTFLOW_VERSION=21.10.6
ARG NEXTFLOW_DOWNLOAD_URL="https://github.com/nextflow-io/nextflow/releases/download/v${NEXTFLOW_VERSION}/nextflow-${NEXTFLOW_VERSION}-all"

RUN cd /usr/local/bin \
    && curl -sL "${NEXTFLOW_DOWNLOAD_URL}" | bash

ENTRYPOINT ["/usr/local/bin/nextflow"]

Running a pipeline:

$ docker build -t nextflow:21.10.6-all -f Dockerfile .
$ docker run --rm -it nextflow:21.10.6-all run brandoncazander/nextflow-file-caching-example -plugins [email protected]
N E X T F L O W  ~  version 21.10.6
Pulling brandoncazander/nextflow-file-caching-example ...
WARN: Cannot read project manifest -- Cause: Remote resource not found: https://api.github.com/repos/brandoncazander/nextflow-file-caching-example/contents/nextflow.config
 downloaded from https://github.com/brandoncazander/nextflow-file-caching-example.git
Launching `brandoncazander/nextflow-file-caching-example` [silly_hawking] - revision: 848da0d07d [master]
Downloading plugin [email protected]
executor >  local (3)
[84/f2922b] process > foo (Fifi) [100%] 3 of 3 ✔

Judging from the nextflow-21.10.6-all contents, I would expect [email protected] to be already included, so I'm not sure why it's downloading it again.