using future on a computing cluster with slurm

[ktyssowski]

I'm trying to use future (with Seurat) on a university computing cluster with slurm. I'm requesting 4-8 cores using the -c flag in my SBATCH script, so they should all be on the same node. However, it does not seem to be able to fork to more than one core. When I run the following:

library(future)

numCores = as.integer(Sys.getenv("SLURM_CPUS_ON_NODE"))
plan(multicore, workers=8)
print(availableCores(methods='mc.cores'))
print(numCores)
print(availableCores("multicore") > 1L)

I get:

current 
      1 
[1] 8
nproc 
FALSE 

so it seems like it's unable to find the correct number of available cores -- is there some reason I should expect this configuration to be incompatible with multicore mode? -- it's not Windows and I'm not using RStudio. Any ideas how to fix it?

[HenrikBengtsson]

What does the following return in such a job?

parallelly::availableCores(which = "all")

Also, what's your sessionInfo(), especially what version of parallelly are you running?

PS. I might have been too quick to move this to Discussions.

[ktyssowski]

If I run this:

Sys.setenv(OMP_NUM_THREADS = 2)
print(parallelly::availableCores(which = "all"))

if it's all on one node, I get this:

system  nproc  Slurm 
    48      2      2 

if i force it to run on two nodes (so one task per node), i get:

system  nproc  Slurm 
    48      1      1 

So it seems like it can only find the cores on a single node (not sure what's expected there), and I never get anything about OpenMP.

ETA: running on a single node with multicores does seem to be working, though!

[HenrikBengtsson]

If I run this:

Sys.setenv(OMP_NUM_THREADS = 2)
print(parallelly::availableCores(which = "all"))

if it's all on one node, I get this:

system  nproc  Slurm 
    48      2      2 

if i force it to run on two nodes (so one task per node), i get:

system  nproc  Slurm 
    48      1      1 

So, just to be clear, that's with the CRAN's parallelly 1.26.0, correct? The develop version, parallelly 1.26.0-9001 should not be affected by OMP_NUM_THREADS as we confirmed above.

So it seems like it can only find the cores on a single node (not sure what's expected there), and I never get anything about OpenMP.

ETA: running on a single node with multicores does seem to be working, though!

Sorry for not being clear in my previous comment: the OpenMP output I mentioned only exists in my head/on the drawing board, right now. I just wanted to give you a heads up in case it's added in the future (pun intended).

Regarding parallelizing across multiple machines. That's a different model, but it is supported via:

plan(cluster)

which defaults to:

plan(cluster, workers = availableWorkers())   ## <= note, not `availableCores()`

Importantly: For that to work, it must be possible to SSH from one compute node to another, which not all HPC clusters support. In contrast, parallelization on a single machine will always work, e.g. plan(multisession), which defaults plan(multisession, workers = availableCores().

If you want to attempt multi-machine parallelization like this, please start a new Discussion thread.

[ktyssowski]

Ooooh sorry I misunderstood!--I thought I was supposed to be seeing the OpenMP info. That's with the develop version, but yeah, I don't think it matters if I set OMP_NUM_THREADS or not (I tried both ways the other day).

And yes, I've been trying the plan(cluster) version. I will open a new thread if I run into issues. Thanks!

[HenrikBengtsson]

That's with the develop version, but yeah, I don't think it matters if I set OMP_NUM_THREADS or not (I tried both ways the other day).

Got it. My bad. Yes, the results actually makes sense with the devel version too. But, as you say, those results are not because you set OMP_NUM_THREADS. They are because Slurm informs cgroups about the number of CPU cores, which in turn affects what nproc returns. That is exactly how we want it to work. Clear as mud?

[ktyssowski]

makes sense! thanks!

[HenrikBengtsson]

The problem here was that availableCores() was affected by environment variable OMP_NUM_THREADS, which may be preset to, say, 1 on some systems, via nproc. In parallelly 1.26.1, which just reached CRAN (2021-06-29), availableCores() is no longer affected by OMP_NUM_THREADS.