disable package startup messages when using future.apply::future_lapply with future::multicore

[elgabbas]

Is it possible to disable package startup messages in future.apply::future_lapply, a behaviour similar to suppressPackageStartupMessages ?
When using future::multicore for HPC jobs submitted via SLURM, I get too many messages into the *.err file—a message for most packages and job combinations. This does not happen for other plans. I am curious if this can be turned off via an argument or by setting an option.
I expect setting stdout=FALSE to disable this, but it may also disable important messages and warnings.
I am using future.apply::future_lapply inside a function, so explicit loading of packages via library/require is not recommended.

Any thoughts?

Thanks in advance

[HenrikBengtsson]

Hi, interesting - haven't considered this use case, but totally valid.

I expect setting stdout=FALSE to disable this, but it may also disable important messages and warnings.

Your concern is valid. One clarification though, in R, functions like cat(), print(), and str() outputs to standard output ("stdout"). You can think of stdout as a piece of paper that becomes filled with output. In contrast, functions like message(), warning(), and error() sends messages via R's condition framework. You can think of conditions as text/SMS messages, which are individual packages of information send somewhere. Eventually those conditions are collected somewhere and then they are displayed one by one.

Future arguments stdout controls whether stdout is relayed or not - the default is stdout = TRUE. Future argument conditions controls which type of conditions are relayed - the default is conditions = "condition" (= all conditions are relayed). In your case, you want to exclude package startup messages from being relayed, but everything else should go trough. Package startup messages are formally of class packageStartupMessage, so that's what we want to exclude, which can be done as:

conditions = structure("conditions", exclude = "packageStartupMessage")

For future_lapply() the corresponding argument names have prefix future., so you want to specify future.conditions as in:

library(future.apply)
plan(multicore)

my_fcn <- function(row_sets) {
  future_lapply(row_sets, FUN = function(rows) {
    # I have to call library() here to get the example to work, 
    # but normally we don't want to do this
    library(mclust)
    cat("Stdout still works\n")
    message("Messages too")
    warning("Warning as well")
    mclust::Mclust(iris[rows, 1:4], verbose = FALSE)
  }, future.conditions = structure("condition", exclude = "packageStartupMessage"))
}

> row_sets <- list(1:50, 51:100, 101:150)
> y <- my_fcn(row_sets)
Stdout still works
Messages too
Stdout still works
Messages too
Stdout still works
Messages too
Warning messages:
1: In ...future.FUN(...future.X_jj, ...) : Warning as well
2: In ...future.FUN(...future.X_jj, ...) : Warning as well
3: In ...future.FUN(...future.X_jj, ...) : Warning as well

PS. The above example was the best I could come up with that emulates what I think you're experiencing. Please try to provide a tiny example when asking for help here or elsewhere - it will save the other person some time and increase your chances for getting a quick reply. I this case I had to go hunt down a package that generated a package startup message.

[HenrikBengtsson]

This does not happen for other plans.

Forgot to comment on this one. It does actually happen to all future backend at least ones. For plan(multisession) you will see it the first time your function is called, but in all following calls, the package is already loaded on the worker, so there will be no more package startup messages produced. In contrast, plan(multicore) starts a fresh parallel worker for each parallel task. This also true for plan(future.callr::callr).

[elgabbas]

Thanks @HenrikBengtsson for your detailed reply.

Unfortunately, this did not work in my case

library(future.apply)
plan(multicore, workers = 10)

my_fcn <- function(rows) {
    # empty function
}

output <- future.apply::future_lapply(
  list(1:50, 51:100, 101:150, 151:200, 1:250, 251:300,
       1:50, 51:100, 101:150, 151:200, 1:250, 251:300,
       1:50, 51:100, 101:150, 151:200, 1:250, 251:300),
  FUN = my_fcn,
  future.conditions = structure("condition", exclude = "packageStartupMessage"),
  future.seed = TRUE, future.packages = c("mclust", "terra"))

Here is the output

Loading required package: future
Package 'mclust' version 6.1.1
Type 'citation("mclust")' for citing this R package in publications.
Package 'mclust' version 6.1.1
Type 'citation("mclust")' for citing this R package in publications.
Package 'mclust' version 6.1.1
Type 'citation("mclust")' for citing this R package in publications.
Package 'mclust' version 6.1.1
Type 'citation("mclust")' for citing this R package in publications.
Package 'mclust' version 6.1.1
Type 'citation("mclust")' for citing this R package in publications.
Package 'mclust' version 6.1.1
Type 'citation("mclust")' for citing this R package in publications.
Package 'mclust' version 6.1.1
Type 'citation("mclust")' for citing this R package in publications.
Package 'mclust' version 6.1.1
Type 'citation("mclust")' for citing this R package in publications.
Package 'mclust' version 6.1.1
Type 'citation("mclust")' for citing this R package in publications.
Package 'mclust' version 6.1.1
Type 'citation("mclust")' for citing this R package in publications.
terra 1.8.51
terra 1.8.51
terra 1.8.51
terra 1.8.51
terra 1.8.51
terra 1.8.51
terra 1.8.51
terra 1.8.51
terra 1.8.51
terra 1.8.51

sessionInfo()
R version 4.4.1 (2024-06-14)
Platform: x86_64-pc-linux-gnu
Running under: SUSE Linux Enterprise Server 15 SP5

Matrix products: default
BLAS/LAPACK: /opt/cray/pe/libsci/24.03.0/GNU/12.3/x86_64/lib/libsci_gnu_mp.so.6.0;  LAPACK version 3.10.1

Random number generation:
 RNG:     L'Ecuyer-CMRG
 Normal:  Inversion
 Sample:  Rejection

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

time zone: Europe/Berlin
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base

other attached packages:
[1] future.apply_1.11.3 future_1.49.0

loaded via a namespace (and not attached):
 [1] BiocManager_1.30.25 compiler_4.4.1      parallelly_1.44.0
 [4] parallel_4.4.1      tools_4.4.1         listenv_0.9.1
 [7] codetools_0.2-20    digest_0.6.37       globals_0.18.0
[10] renv_1.1.4

[HenrikBengtsson]

I see what's going on. The exclude = "packageStartupMessage" takes place after packages in future.packages are loaded. This is something I can fix in the next release of the future package.

Now, as a workaround, which is actually what I prefer, because it avoids the visual and mental clutter of having to explicitly declare future.packages, is to "guide" futureverse using mclust::... in the code, even if just as a no-op, e.g.

my_fcn <- function(rows) {
    # hint that 'mclust' is needed
    mclust::map
    
    # empty function
}

If you then drop mclust from future.packages, it will be loaded when my_fcn() is called, and then packageStartupMessage will be dropped.

[elgabbas]

Thanks @HenrikBengtsson. I always use :: to load functions from a specific package. But I think I had errors in some situations, unless I clearly provide the list of packages to the future.packages argument. Thanks for fixing this in the new version of the package.

[HenrikBengtsson]

I've fixed this in the next release of future; [future.]conditions = structure("condition", exclude = "packageStartupMessage") should now be respected also for packages loaded and attached via the [future.]packages argument.

Please retry with:

remotes::install_github("HenrikBengtsson/future", ref = "develop")

[HenrikBengtsson]

future 1.58.0 fixing this is now on CRAN

[elgabbas]

Great. Thanks @HenrikBengtsson