furrr::future_map_df, max.concurrent.jobs for torque cluster

[richardbeare]

Hi,
I'm not sure what I'm missing here. I have a problem that I want to pack into jobs on a pbs cluster with a size that can run within a two hour time limit. This will result in some thousands of jobs. Per user cluster limits prohibit that many jobs from being submitted simultaneously. The call generating the jobs is of this form:

result <- future_map_dfr(chunks, ~fitfunction(.x, ...))

The batchtools option of max.concurrent.jobs appears to address this issue, but I can't seem to set it via plan or config file, and haven't found any examples.

Attempts:
a batchtools.conf.R file containing

max.concurrent.jobs <- 200

Is that the right format?

adding registry to the plan call:

plan(batchtools_torque, workers=10000, template="batchtools.pbs.tmpl", 
resources=list(nodes="1:ppn=4", walltime="2:00:00", pmem="6G"),
registry=list(max.concurrent.jobs = 30L))

I can see the submitJobs code is doing:

  max.concurrent.jobs = assertCount(resources$max.concurrent.jobs, null.ok = TRUE) %??%
    assertCount(reg$max.concurrent.jobs, null.ok = TRUE) %??% NA_integer_

So the registry looks like it is the right place to put this, but what is the right way of doing this from furrr or future_map?

I appreciate any advice

[HenrikBengtsson]

This will result in some thousands of jobs. Per user cluster limits prohibit that many jobs from being submitted simultaneously.
...

plan(batchtools_torque, workers=10000, template="batchtools.pbs.tmpl", ...)

max.concurrent.jobs <- 200

I don't know whether that's correct or if it'll work. I simply don't know enough of the ins and outs of batchtools to say something useful. I think it's best to ask on https://github.com/mlr-org/batchtools/issues to see how it should work and if it works.

BTW, there's also mlr-org/batchtools#235, which says "batchtools will try to submit any jobs passed to submitJobs() as fast as possible (but sequentially)". That one is from 2019, so it could be that it is no longer valid and should be closed.

An alternative, second best, approach is to limit the total number of workers that the future ecosystem sees:

plan(batchtools_torque, workers=200, template="batchtools.pbs.tmpl", ...)

With this approach, result <- future_map_dfr(chunks, ~fitfunction(.x, ...)) will chunk up the chunks into 200 equally sized tasks and submit those as 200 jobs.

Now, if you don't want to chunk up chunks, but have each element in chunks to be a separate jobs (= a separate future), then you can control that by chunk_size = 1L, e.g.

result <- future_map_dfr(chunks, ~fitfunction(.x, ...), .options = furrr_options(chunk_size = 1L))

That will resolve each iterator one-by-one in a separate future, which means one job per iteration to the job scheduler. Because we set workers = 200 in the plan, the future ecosystem will allow for at most 200 active futures. When we get to future 201, it will not be submitted to the scheduler until at least one of the 1-200 futures have finished.

FWIW, using the new futurize package (2026-01-22), you can do:

plan(batchtools_torque, workers=200, template="batchtools.pbs.tmpl", ...)

result <- purrr::map_dfr(chunks, ~fitfunction(.x, ...)) |> futurize(chunk_size = 1L)

to achieve the exact same thing.