Problem with starting multiple Julia process on a cluster at the same time

[newptcai]

I am trying to start multiple Julia processes on a cluster at the same time using a Python script (parallel-ssh). I noticed that, a few of these processes will fail to start, with the following errors.

The cluster has a shared network file system. This could be the source of the issue. But this has not been any problem when previously everything is done in Python.

ERROR: LoadError: SystemError: opening file "/home/myname/.julia/compiled/v1.1/MyPackage/ttkBW.ji": No such file or directory
Stacktrace:
 [1] #systemerror#43(::Nothing, ::Function, ::String, ::Bool) at ./error.jl:134
 [2] systemerror at ./error.jl:134 [inlined]
 [3] #open#309(::Bool, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Function, ::String) at ./iostream.jl:283
 [4] #open at ./none:0 [inlined]
 [5] open(::String, ::String) at ./iostream.jl:339
 [6] stale_cachefile(::String, ::String) at ./loading.jl:1321
 [7] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:693
 [8] _require(::Base.PkgId) at ./loading.jl:937
 [9] require(::Base.PkgId) at ./loading.jl:858
 [10] require(::Module, ::Symbol) at ./loading.jl:853
 [11] include at ./boot.jl:326 [inlined]
 [12] include_relative(::Module, ::String) at ./loading.jl:1038
 [13] include(::Module, ::String) at ./sysimg.jl:29
 [14] exec_options(::Base.JLOptions) at ./client.jl:267
 [15] _start() at ./client.jl:436
in expression starting at /home/myname/code/MyPackage.jl/src/gwsim.jl:3

[newptcai]

I added 0.1 sleep time between starting each process. This seems to have avoided the problem.

[vchuravy]

related issue #30174

[WT215]

I added 0.1 sleep time between starting each process. This seems to have avoided the problem.

Hi, @newptcai , I have similar issue when I deployed julia in HPC for running jobs. Sometimes the package ``KernelDensity'' can be loaded in a worker correctly but sometimes failed.

How did you add sleep time between starting each process?

Thanks in advance.

[simonbyrne]

This is still an issue on 1.7.2:
https://buildkite.com/clima/climaatmos-ci/builds/864#babf55c9-1a81-4228-9a61-163128f0e1dd

My guess as to what is happening is that one process is attempting to load a precompile package, while another deletes it from the cache. In particular, it gets deleted between this line
https://github.com/JuliaLang/julia/blob/v1.7.2/base/loading.jl#L1087
and this line
https://github.com/JuliaLang/julia/blob/v1.7.2/base/loading.jl#L1097

There are a couple of options as far as I can see:

1. some sort of locking to prevent deletion while loading: we now have a Pidfile support in the Filewatching stdlib (Add Pidfile to FileWatching #44367), but unfortunately this code is in Base so it's not clear how we could make use of it.
2. Allow _require_search_from_serialized to fail gracefully if it can't open the file.

[IanButterworth]

master now has pidlocking around the precompilation process

[simonbyrne]

Appears to be PRs #49052, #50214 & #50254