Skip to content

Commit e1cc26f

Browse files
Update docs with information on the CPU-overuse protection error [#745]
1 parent 8098bb5 commit e1cc26f

File tree

3 files changed

+27
-2
lines changed

3 files changed

+27
-2
lines changed

DESCRIPTION

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
Package: future
2-
Version: 1.34.0-9125
2+
Version: 1.34.0-9126
33
Title: Unified Parallel and Distributed Processing in R for Everyone
44
Imports:
55
digest,

NEWS.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,12 @@
88
`plan()`. Previously, this had to be implemented by each backend,
99
but now it's handled automatically by the future framework.
1010

11+
## Documentation
12+
13+
* Updated the future topology vignette with information on the
14+
CPU-overuse protection error that may occur when using a nested
15+
future plan and how to avoid it.
16+
1117

1218
# Version 1.34.0 [2024-07-29]
1319

vignettes/future-3-topologies.md.rsp

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,26 @@ Now, we could imagine that we process the outer layer with, say, two parallel fu
105105
plan(list(tweak(multisession, workers = 2), tweak(multisession, workers = I(4))))
106106
```
107107

108-
When using this approach, there is a risk of setting up too many concurrent workers. Because Futureverse has a built-in protection, we need to declare nested workers using the As-Is `I(.)` function, which basically tells the parallel framework "trust us, we know what we are doing". To minimize the risk of mistakes and to make sure our setup respects `availableCores()`, use something like:
108+
Note that As-Is `I(.)` specification for the inner layer, i.e. `workers = I(4)`. If we would just specify `workers = 4`, the future framework would detect this as a potential user mistake. This is because by default it prevents nested parallelization and allots only a single CPU core to the inner layer, i.e. `availableCores()` will return one there. However, the user requests four CPU cores, which could result in an unintended 400% CPU overuse. The future framework detects this discrepancy, and if it is too large, it will produce an error. For example, on an eight core machine, we would get the following error produced at the inner layer:
109+
110+
```sh
111+
> plan(list(tweak(multisession, workers = 2), tweak(multisession, workers = 4)))
112+
> a %<-% { b %<-% 1 ; b }
113+
> a
114+
Error in checkNumberOfLocalWorkers(workers) :
115+
Attempting to set up 4 localhost parallel workers with only 1 CPU
116+
cores available for this R process (per ‘mc.cores’), which could
117+
result in a 400% load. The hard limit is set to 300%. Overusing the
118+
CPUs has negative impact on the current R process, but also on all
119+
other processes of yours and others running on the same machine. See
120+
help("parallelly.maxWorkers.localhost", package = "parallelly") for
121+
further explanations and how to override the hard limit that triggered
122+
this error
123+
```
124+
125+
Because Futureverse has this built-in protection, we need to explicitly override it by declaring nested workers using the As-Is `I(.)` function. This basically tells the parallel framework "trust us, we know what we are doing". To minimize the risk of mistakes and to make sure our setup respects `availableCores()`.
126+
127+
To make sure we stay within the limits of the current machine, it's best to use something like:
109128

110129
```r
111130
plan(list(

0 commit comments

Comments
 (0)