Can't lock file history & Can't update history file

[Nathanrown]

Bug report

When I use a script to run an array of jobs simultaneously , some of them got Can't lock file: /nextflow/.nextflow/history -- Nextflow needs to run in a file system that supports file locks error message.
and I got error Can't update history file: .nextflow/history when I run the script again.
Then it seems nextflow couldn't work anymore unless I revised the history file.

Expected behavior and actual behavior

All pipelines finished.

Steps to reproduce the problem

Run the following script.

for num in `seq 1 20`
do
{
        nextflow run tutorial.nf
} &
done

file tutorial.nf got from https://www.nextflow.io/docs/latest/getstarted.html#your-first-script

Program output

(Copy and paste here output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the .nextflow.log file.)

Environment

• Nextflow version: [22.10.6]
• Java version: [11]
• Operating system: [Linux]
• Bash version: (use the command $SHELL --version)

Additional context

(Add any other context about the problem here)

[bentsherman]

You can't run multiple pipelines concurrently in the same directory. Each pipeline will try to acquire a lock on the .nextflow directory, which contains the cache database. If you want to run them in parallel, you need to run each pipeline in its own directory.

[Nathanrown]

you mean each pipeline has its own history file?
but if I want to get all pipelines I had run with nextflow log how I can do for that.

[bentsherman]

A history file can hold multiple pipeline runs, it's just that it can only be edited by one pipeline at a time. It's not like a MySQL server that can handle multiple clients in parallel, it's just a database file. So if you want to run multiple pipelines at the same time, you will have to run them in different directories and they will maintain separate history files.

If you want to maintain a single history while also running many pipelines in parallel, that is one of the main use cases of Tower. You should check it out if you haven't already!

[Nathanrown]

Thank you for your patience help.