Improve Nextflow resilience to eventually consistent FileSystems

[BioWilko]

This PR does the following:

• Move all .exitcode checking logic into a generic ExitStatusAwaiter func which will retry on the next polling loop for API based executors, this is a re-implementation of the existing Grid logic.
• Modify all (non-local) executors to utilise this, including plugins where appropriate
• Create a new TaskFileRetry.groovy generic blocking retry wrapper which ensures files exist, currently used to check for .command.env and .command.out only
• Use of the executor.exitReadTimeout param should be consistent across executors for this purpose.
• Add ceph to Shared FS detection
• Modify the sync cmd from sync || true to sync --file-system /work/dir || true -> Might reduce impact on HPCs hopefully
• Move sync_cmd from after cleanup_cmd to immediately before .exitcode writing so that all workdir writes are flushed before .exitcode is written
• Default sync_cmd to on in shared FS -> I understand this could be quite controversial, happy to change this

What hasn't been done:

• Docs haven't been updated
• Core plugin changelogs / versions haven't been updated, I wasn't sure what internal processes were inplace to do this...

This PR fixes #7247 which I've confirmed is due to eventually consistent filesystems causing problems stochastically, I've tested it on a production K8S system as fully working but I don't have access to others for a full test.

[netlify]

✅ Deploy Preview for nextflow-docs canceled.

Name	Link
🔨 Latest commit	0e863b6
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs/deploys/6a56487f48357e00086387fc

[BioWilko]

@bentsherman any chance somebody on your end might take a look at this?

If it makes it easier I could always split the changes into a set of more atomic PRs? e.g.

• Change to generic ExitStatusAwaiter and update core plugins
• Change to general .command.env awaiting logic
• Change to sync_cmd and isSharedFS helper for ceph

[BioWilko]

I can't reproduce these failures locally so suspect there's something screwy with the CI...

[bentsherman]

Looks like a compilation error:

/home/runner/work/nextflow/nextflow/plugins/nf-azure/src/test/nextflow/cloud/azure/batch/AzBatchTaskHandlerTest.groovy: 233: unable to resolve class BatchTaskExecutionInfo
> Task :plugins:nf-azure:compileTestGroovy FAILED
 @ line 233, column 24.
           def execInfo = new BatchTaskExecutionInfo(0,0)
                          ^

Your PR does edit that file but not sure how it affects this line. So you aren't getting this error locally?

Maybe try updating against latest master?

[BioWilko]

You're right, it was a basic missing import error!

[riederd]

I build the PR locally and run it with one of our pipelines in which the test input data leads to many short lived processes. I use slurm and cephfs. Unfortunately the pipeline is still often waiting forever that jobs are finished, which they actually already are but it is not noticed by nextflow. Only after running find <workdir> > /dev/null manually, nextflow sees the jobs as finished and continues.

[BioWilko]

I build the PR locally and run it with one of our pipelines in which the test input data leads to many short lived processes. I use slurm and cephfs. Unfortunately the pipeline is still often waiting forever that jobs are finished, which they actually already are but it is not noticed by nextflow. Only after running find <workdir> > /dev/null manually, nextflow sees the jobs as finished and continues.

Does it do the same on release Nextflow? Or only on my PR version?

[riederd]

I build the PR locally and run it with one of our pipelines in which the test input data leads to many short lived processes. I use slurm and cephfs. Unfortunately the pipeline is still often waiting forever that jobs are finished, which they actually already are but it is not noticed by nextflow. Only after running find <workdir> > /dev/null manually, nextflow sees the jobs as finished and continues.

Does it do the same on release Nextflow? Or only on my PR version?

It does it also on the release Nextflow, but it is even worse there. I tried to get an answer from claude to explain the situation, this is what I got:

1. Nextflow submits a burst of short jobs to Slurm. They all finish quickly and write .exitcode to CephFS, then release their filesystem caps back to the MDS.

2. The head node already holds a directory listing capability (Ls cap) that it got before the jobs ran. Because no other client is contending for a conflicting cap, the MDS has no reason to revoke it.

3. When Nextflow polls, it calls exists() or listDirectory() on the work directories. The CephFS kernel client on the head node sees it still holds a valid Ls cap and serves the answer from its local cache — which predates the .exitcode files being written. The files appear not to exist.

4. This is not a timing issue with a TTL — unlike NFS where stale entries expire after a fixed timeout, CephFS negative dentries live as long as the cap is valid, which is indefinitely if there's no contention. So Nextflow waits forever.

5. find work/ fixes it because it issues stat() calls on individual paths, which bypass the directory listing cache and go directly to the MDS, which returns the current truth.

The core mismatch is that CephFS is designed around contention-driven consistency: caches are only invalidated when clients fight over caps. Nextflow's polling is purely passive — it never creates contention, so it never gets the cache invalidated, and never sees the completed jobs.

Not sure if this is correct, though

[BioWilko]

Okay I'm glad that my PR doesn't introduce more issues, I don't think I'm sufficiently informed about network filesystems to be sure but Nextflow actively checks the .exitcode file when an API based executor (like slurm) finishes a process, the find fixing it seems very strange as if your cephfs client is misconfigured or something...

[riederd]

Okay I'm glad that my PR doesn't introduce more issues, I don't think I'm sufficiently informed about network filesystems to be sure but Nextflow actively checks the .exitcode file when an API based executor (like slurm) finishes a process, the find fixing it seems very strange as if your cephfs client is misconfigured or something...

my cephfs client mount options (kernel client) are pretty much standard no special configuration there:

secretfile=/etc/ceph/ceph.client.cephfs_user.secret,wsync,rw,noatime,_netdev,recover_session=clean

Maybe you have an idea where to introduce some debug lines to see where it's hanging