PID/Hostname File Locking for Cluster Environments.

[willow-ahrens]

The PID file locking ecosystem in Python may not be robust to the networked filesystems common to HPC clusters, which may have multiple hosts on multiple operating systems.

Quoting @amilsted:

Julia core uses https://github.com/vtjnash/Pidfile.jl to do hostname/PID-based locking to JuliaLang/julia#42405 and JuliaLang/julia#42405, because this type of locking is JuliaLang/julia#31953.

A successful fix for this issue would identify and integrate a python pidfile implementation which offers the same (tested) guarantees as https://github.com/vtjnash/Pidfile.jl. Further discussion in #45, where FileLock may be used as an interim solution.

[nilsdeppe]

Hi! Just to follow up on this, I've run into this same issue and it basically would mean we can't use Julia at all in a shared environment (read: I have to tell someone they have to port thousands of lines of Julia code into Python or C++). What's not clear to me is why the pidlock is used even when I set export PYTHON_JULIAPKG_OFFLINE=yes. I guess someone could write simultaneously while I only read, but that seems like it's basically my own fault. Anyway, would it be possible to add an env variable that would disable the pidlock check completely for those of us who don't want it? As it stands, one user installs everything on the filesystem that we need, makes the entire directly read-only, and then nobody should ever write there again. In this case (since nobody has write permissions to the entire python and julia environement directory), we have a strong guarantee that nobody is writing while we are reading, and so the pidlock is just a burden.

Having said all that, it's still not at all clear to me that Julia has any great way to share not just the core executable, but also all installed dependencies, so is trying that essentially bound to fail?

Thank you!

Edit: it also seems like the lock.pid is not being cleaned up even after I exit the python session, whereas I assume it should be cleaned up as soon as juliapkg has resolved.

[cjdoris]

@nilsdeppe yep something along those lines seems reasonable, I've opened a separate issue about it.