the R function mclapply() in the parallel package doesn't seem to work when rendering a quarto document

[paulgribble]

Description

The following R code when placed within a quarto document and rendered with quarto preview only uses 1 core even when instructed to use 16 cores (my machine has 16 cores).

The identical code in an RStudio or R session does use all 16 cores.

Is this a limitation somehow of quarto?

library(parallel)
work <- function(n) {
  ng <- 3
  df <- tibble(x=c(rnorm(ng,100,15), rnorm(ng,100,15), rnorm(ng,100,15)), 
               g=factor(c(rep("group1",ng), rep("group2",ng), rep("group3",ng))))
  summary(aov(x~g, data=df))[[1]]$"F value"[1]
}
nsims = 10000
ng = 3
n = rep(10, nsims)
results <- mclapply(n, work, mc.cores=16)
fvals <- rep(0, nsims)
for (i in 1:nsims) {
  fvals[i] <- results[[i]]  
}
fvals <- tibble(fvals)

[mcanouil]

Did you try within an Rmarkdown document?
If you observed the same behaviour, it means it's not Quarto.

---
output: html_document
---

```{r}
library(parallel)
work <- function(n) {
  ng <- 3
  df <- tibble(x=c(rnorm(ng,100,15), rnorm(ng,100,15), rnorm(ng,100,15)), 
               g=factor(c(rep("group1",ng), rep("group2",ng), rep("group3",ng))))
  summary(aov(x~g, data=df))[[1]]$"F value"[1]
}
nsims = 10000
ng = 3
n = rep(10, nsims)
results <- mclapply(n, work, mc.cores=16)
fvals <- rep(0, nsims)
for (i in 1:nsims) {
  fvals[i] <- results[[i]]  
}
fvals <- tibble(fvals)
```

[cscheid]

We shouldn't expect people to know rmarkdown in other to diagnose things about Quarto.

[mcanouil]

Fair point. (although very unlikely an R user today does not know anything about rmarkdown)

[cscheid]

even when instructed to use 16 cores

How did you do this instruction? Quarto has no special parallel handling; I think what's happening is that your configuration is not getting communicated to the library.

[paulgribble]

OK I tried it in RMarkdown and it successfully uses multiple cores. The same code in a quarto document when rendered or previewed does not.

even when I hard code the number of cores by calling results <- mclapply(n, work, mc.cores=16) instead of using the detected number of cores, when rendered using Quarto it still only uses 1 core.

I guess this means it is in fact somehow Quarto's fault. Strange

Here is the code

library(tidyverse)
library(parallel)
work <- function(n) {
  ng <- 3
  df <- tibble(x=c(rnorm(ng,100,15), rnorm(ng,100,15), rnorm(ng,100,15)), 
               g=factor(c(rep("group1",ng), rep("group2",ng), rep("group3",ng))))
  summary(aov(x~g, data=df))[[1]]$"F value"[1]
}
nsims = 10000
ng = 3
n = rep(10, nsims)
numCores <- detectCores()
results <- mclapply(n, work, mc.cores=numCores)
fvals <- rep(0, nsims)
for (i in 1:nsims) {
  fvals[i] <- results[[i]]  
}
fvals <- tibble(fvals)
ggplot(fvals, aes(x=fvals)) + geom_histogram(bins=50)

[mcanouil]

Could you try something simple? As I am not able to reproduce your described behaviour.
Your 10,000 processes are very small. By the way, not a good idea to fork parallel like that on small tasks. You spend more time creating the forked R processes and retrieving the results than computing the results.

---
format: html
---

```{r}
library(parallel)
n_cores <- 16
system.time({
  results <- mclapply(
    X = seq_len(n_cores),
    FUN = function(n) {
      Sys.sleep(60)
    },
    mc.cores = n_cores
  )
})
```

With the above code, if it runs in parallel, the elapsed time will be 60 seconds, otherwise 16 times 60 seconds.

10 Cores	1 Core

[paulgribble]

I quarto preview'd in VSCode with your code above and it was indeed 60 sec so it was using all cores. Strange.

thanks for the help I think I know what to do