some jobs among a slurm job array fail to addprocs

[TimeExplorer]

A job array with identical input parameters is submitted to slurm cluster. While a number of jobs among them are successfully finished, most of others just fail at the beginning at the expression

addprocs_slurm(parse(Int, ENV["SLURM_NTASKS"]))

with the following message:

ERROR: LoadError: TaskFailedException:
IOError: unlink: no such file or directory (ENOENT)
Stacktrace:
[1] uv_error at ./libuv.jl:97 [inlined]
[2] unlink(::String) at ./file.jl:885
[3] rm(::String; force::Bool, recursive::Bool) at ./file.jl:268
[4] rm at ./file.jl:260 [inlined]
[5] iterate at ./generator.jl:47 [inlined]
[6] _collect(::Array{String,1}, ::Base.Generator{Array{String,1},typeof(rm)}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:678
[7] collect_similar at ./array.jl:607 [inlined]
[8] map at ./abstractarray.jl:2072 [inlined]
[9] launch(::SlurmManager, ::Dict{Symbol,Any}, ::Array{WorkerConfig,1}, ::Base.GenericCondition{Base.AlwaysLockedST}) at /u/guoyi/.julia/packages/ClusterManagers/7pPEP/src/slurm.jl:39
[10] (::Distributed.var"#39#42"{SlurmManager,Dict{Symbol,Any},Array{WorkerConfig,1},Base.GenericCondition{Base.AlwaysLockedST}})() at ./task.jl:358
Stacktrace:
[1] wait at ./task.jl:267 [inlined]
[2] addprocs_locked(::SlurmManager; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:494
[3] addprocs_locked at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:448 [inlined]
[4] addprocs(::SlurmManager; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:441
[5] addprocs at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:435 [inlined]
[6] #addprocs_slurm#15 at /u/guoyi/.julia/packages/ClusterManagers/7pPEP/src/slurm.jl:85 [inlined]
[7] addprocs_slurm(::Int64) at /u/guoyi/.julia/packages/ClusterManagers/7pPEP/src/slurm.jl:85

[dominikkiese]

I had the same issue a while ago. I could resolve this, by, instead of using a job array, submitting everything as single jobs which all had a different working directory. Its certainly not an elegant solution, and certainly not what one is intended to do, but it worked at least. I am not sure, but I think there is an issue with multiple workers writing their verbose output to the same file ...

[juliohm]

@TimeExplorer I will close the issue because I can't understand what you tried to do. Please provide a MWE or more explanation about how you are trying to submit the jobs.