Detect a future parallel backend

[talegari]

As a package developer, I need to know if user is using a parallel backend or not. Is there a better way than parallel_flag = !("sequential" %in% class(future::plan())) ?

[scottkosty]

You can see here for one approach.

By the way, I'm curious about your specific use case. Why, as a package developer, do you need to know if the user is using a parallel backend or not?

[HenrikBengtsson]

I'd like to emphasize @scottkosty 's question/comment:

Why, as a package developer, do you need to know if the user is using a parallel backend or not?

This is most important, because it's at the very core of Futureverse design to separate the developer's and the user's responsibilities. Knowing what you're trying to achieve, will help give you a more constructive answer.

There's also nbrOfWorkers(). Note also that "parallel" can mean many things. For instance, plan(cluster, worker = 1) runs in parallel, but with a single background process, so nbrOfWorkers() == 1.

PS. Please use inherits(x, class_name) instead of class_name %in% class(x) to test for inheritance in R. See for example https://blog.r-project.org/2019/11/09/when-you-think-class.-think-again/.

[talegari]

I understand what I am asking for seems antithetic to design of future to let developer not worry if user uses parallel backend or not.

This is my specific requirement: Using parallel backend for some data.table operations. The way data.table works, there seems to be little benefit (or even negative) of using parallel backend unless the data and number of groups are large enough.

Rough pseudocode:

if not parallel:
   dt[, , group = "a"] # typical DT operation
else:
   dt |> split(by = "a") |> future.apply::future.lapply(foo) |> rbindlist()

The catch is, for sequential, the if part (not parallel) is much more faster than else part. Hence, I cannot allow future.lapply to fallback to lapply.