github repo lock conflict

[lstrgar]

Hi. I am launching multiple Nextflow processes that run in parallel. Occasionally I'll see a job fail with the following error message:

Mar-23 15:36:55.481 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 21.10.5
Mar-23 15:36:56.536 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/.nextflow/assets/labsyspharm/mcmicro/.git/config; branch: master; remote: origin; url: https://github.com/labsyspharm/mcmicro.git
Mar-23 15:36:56.586 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/.nextflow/assets/labsyspharm/mcmicro/.git/config; branch: master; remote: origin; url: https://github.com/labsyspharm/mcmicro.git
Mar-23 15:36:57.238 [main] DEBUG nextflow.cli.Launcher - Operation aborted
org.eclipse.jgit.api.errors.JGitInternalException: Cannot lock /home/.nextflow/assets/labsyspharm/mcmicro/.git/index. Ensure that no other process has an open file handle on the lock file /home/.nextflow/assets/labsyspharm/mcmicro/.git/index.lock, then you may delete the lock file and retry.
        at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:334)
        at nextflow.scm.AssetManager.checkout(AssetManager.groovy:927)
        at nextflow.cli.CmdRun.getScriptFile(CmdRun.groovy:381)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:264)
        at nextflow.cli.Launcher.run(Launcher.groovy:480)
        at nextflow.cli.Launcher.main(Launcher.groovy:639)
Caused by: org.eclipse.jgit.errors.LockFailedException: Cannot lock /home/.nextflow/assets/labsyspharm/mcmicro/.git/index. Ensure that no other process has an open file handle on the lock file /home/.nextflow/assets/labsyspharm/mcmicro/.git/index.lock, then you may delete the lock file and retry.
        at org.eclipse.jgit.dircache.DirCache.lock(DirCache.java:252)
        at org.eclipse.jgit.dircache.DirCache.lock(DirCache.java:329)
        at org.eclipse.jgit.dircache.DirCache.lock(DirCache.java:295)
        at org.eclipse.jgit.lib.Repository.lockDirCache(Repository.java:1282)
        at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:268)
        ... 5 common frames omitted

These jobs are being run in an HPC environment with a scheduler, so there is a potential queueing delay that makes a simple solution like delaying the launch of each process by a fixed duration not robust.

Is there a principled way of launching multiple Nextflow jobs in parallel to avoid the issues of a shared Github repository? It's not exactly clear to me what operations exactly require obtaining a lock, but if anyone was able to share information about this it might be helpful.

Thanks in advance.

[pditommaso]

This is a bit unusual, what's the command used to launch the execution

[lstrgar]

Here is the command:

nextflow -log $LOGFILE run labsyspharm/mcmicro -r $GITHASH --in $DATADIR -w $WORKDIR -c $CFG -profile singularity --start-at illumination --stop-at registration -params-file $OPTS_FILE

The workflow is all contained here:

https://github.com/labsyspharm/mcmicro

I can provide additional info (e.g. specific githash or config files) but I don't suspect they'll have anything to do with the issue. The config file only provides directions to the executor (in this case SLURM) to allow more than the default memory allocation. The opts file are command line options to the programs mcmicro is composed out of.

My original suspicion was maybe that there are some git operations occurring because of the -r specifying a particular git hash, which is not the latest.