task dir not locked across concurrent top-level nextflow instances

[notestaff]

When Nextflow starts a task, it locks the task execution directory under the workDir:
https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy#L777
But this lock (if I understand correctly) only works within the current top-level Nextflow invocation.
If another top-level Nextflow instance is started concurrently with the same workDir, and has a task with the same hash, could this result in two tasks writing to the same task dir, potentially clobbering the output?

More generally, it would help if Nextflow's official documentation could clarify whether such concurrent execution is supported. I was informally told that the workDir can be shared but not the launchDir, but what is the official spec? Thanks a lot!

[bentsherman]

I agree it might be worthwhile to add a blurb about this to the docs.

AFAIK, you were told correctly. You can't really have two concurrent Nextflow sessions in the same directory because they will clobber the .nextflow directory which contains cache metadata. However, sharing the work directory is 99.9999...% safe. It is extremely unlikely that the two Nextflow sessions would generate tasks with the same hash. In fact, it might even be impossible if the hash is based on the session ID, but I would need to verify that from the codebase.

[jmuhlich]

I don’t think this issue has been mentioned in the documentation yet so I just want to keep this from going stale.

Would it be possible to check for a concurrent session at startup? There is obviously a race condition so it wouldn’t catch every single instance, but it would probably help a lot. Our users on an HPC cluster get burned by this frequently.

[bentsherman]

I have converted this issue into a discussion so that it doesn't go stale and can be found more easily by other users. Multiple Nextflow runs can share the same work directory but not the same launch directory. Tasks are guaranteed to be unique even across multiple concurrent Nextflow runs because the session ID is included in the task hash.