MPI clusters not supported since future 1.34.0 [solved]

[helijuottonen]

Hi, I'm looking for any advice on how to use future::plan(cluster) on multiple nodes in a Slurm-based HPC environment. Previously (until future v. 1.34.0) we used MPI clusters that were formed with snow and RMPISNOW at the start of the job, so that the following worked:
cl <- snow::getMPIcluster()
plan(cluster, workers = cl)

Since this does not work anymore in future after the recent changes, I would be grateful for any leads on if using pre-created MPI clusters is still possible in future 1.58.0 and how they should be passed to future, and in general what is the recommend way to use future::plan(cluster) in HPC multinode jobs.

Some additional information: our HPC cluster does allow ssh between nodes, but it seems it is not straightforward to get the socket cluster approach to work (still working on it). I'm aware of future.batchtools, but tools that launch jobs within jobs are complicated for resource distribution and we would prefer to avoid these approaches if possible.

[HenrikBengtsson]

... we used MPI clusters that were formed with snow and RMPISNOW at the start of the job, so that the following worked: cl <- snow::getMPIcluster() plan(cluster, workers = cl)

Since this does not work anymore in future after the recent changes

I cannot see what have changed that would make it no longer work. Can you please clarify what is not longer working? What happens if you use it? Do you get an error message? What does print(cl) output? What version of future and R are you using?

[HenrikBengtsson]

... run our test example that runs a function on 3 files with future_map, I get a seg fault (error message below).

Can you please rerun with:

trace(Rmpi::mpi.iprobe, tracer = quote({ utils::str(list(method = "mpi.iprobe()", source = source, tag = tag, comm = comm, status = status)) }), print = FALSE)

and share those trace outputs?

[helijuottonen]

Trace output with future 1.58.0 + workaround line + seg fault:

[1] "mpi.iprobe"
List of 5
 $ method: chr "mpi.iprobe()"
 $ source: NULL
 $ tag   : int -1
 $ comm  : NULL
 $ status: num 0

Trace output with future 1.34.0 that works as expected:

[1] "mpi.iprobe"
List of 5
 $ method: chr "mpi.iprobe()"
 $ source: int 1
 $ tag   : int -1
 $ comm  : num 0
 $ status: num 0
List of 5
 $ method: chr "mpi.iprobe()"
 $ source: int 2
 $ tag   : int -1
 $ comm  : num 0
 $ status: num 0
List of 5
 $ method: chr "mpi.iprobe()"
 $ source: int 3
 $ tag   : int -1
 $ comm  : num 0
 $ status: num 0

(repeated many times)

[HenrikBengtsson]

Thanks for this info; I've fixed another mistake. Please retry with future (>= 1.67.0-9003);

remotes::install_github("futureverse/future", ref="develop")

[helijuottonen]

Done, and now everything works as expected. Thank you so much!

[HenrikBengtsson]

Excellent. Thank you for reaching out about this problem!

[HenrikBengtsson]

future 1.68.0, which fixes this issue, is now on CRAN.