findEncodingDim always uses all avaialble cores

[emariella]

Dear OUTRIDER authors,

thank you for this very useful package. These days i am doing some initial tests and unfortunately I am not able to limit the number of cores that are used by the findEncodingDim function: it ignores my BiocParallel setting, thus always using all avaialble cores. I'm having a hard time finding my mistake and unfortunately it is a very serious problem for me because I currently work on a shared server.

This is the simple code that I am running:

ncores <- 10
register(MulticoreParam(ncores))
ods <- findEncodingDim(ods, BPPARAM=bpparam())

I really hope you can help me. I also attach the session info, please let me know if you need additional information.

Thanks in advance!

Elisa

> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-conda_cos6-linux-gnu (64-bit)
Running under: Ubuntu 18.04.4 LTS

Matrix products: default
BLAS/LAPACK: /scratch/Elisa/miniconda3/envs/rna-seq/lib/R/lib/libRblas.so

locale:
 [1] LC_CTYPE=en_US.utf8       LC_NUMERIC=C             
 [3] LC_TIME=en_US.utf8        LC_COLLATE=en_US.utf8    
 [5] LC_MONETARY=en_US.utf8    LC_MESSAGES=en_US.utf8   
 [7] LC_PAPER=en_US.utf8       LC_NAME=C                
 [9] LC_ADDRESS=C              LC_TELEPHONE=C           
[11] LC_MEASUREMENT=en_US.utf8 LC_IDENTIFICATION=C      

attached base packages:
[1] stats4    parallel  stats     graphics  grDevices utils     datasets 
[8] methods   base     

other attached packages:
 [1] tximport_1.14.2             DESeq2_1.26.0              
 [3] OUTRIDER_1.4.2              data.table_1.12.8          
 [5] SummarizedExperiment_1.16.1 DelayedArray_0.12.3        
 [7] matrixStats_0.56.0          GenomicFeatures_1.38.2     
 [9] AnnotationDbi_1.48.0        Biobase_2.46.0             
[11] GenomicRanges_1.38.0        GenomeInfoDb_1.22.1        
[13] IRanges_2.20.2              S4Vectors_0.24.4           
[15] BiocGenerics_0.32.0         BiocParallel_1.20.1        

loaded via a namespace (and not attached):
  [1] colorspace_1.4-1         ellipsis_0.3.0           htmlTable_1.13.3        
  [4] XVector_0.26.0           base64enc_0.1-3          rstudioapi_0.11         
  [7] bit64_0.9-7              codetools_0.2-16         splines_3.6.1           
 [10] PRROC_1.3.1              geneplotter_1.64.0       knitr_1.28              
 [13] Formula_1.2-3            jsonlite_1.6.1           Rsamtools_2.2.3         
 [16] annotate_1.64.0          cluster_2.1.0            dbplyr_1.4.2            
 [19] png_0.1-7                pheatmap_1.0.12          compiler_3.6.1          
 [22] httr_1.4.1               backports_1.1.5          assertthat_0.2.1        
 [25] Matrix_1.2-17            lazyeval_0.2.2           cli_1.1.0               
 [28] acepack_1.4.1            htmltools_0.4.0          prettyunits_1.1.1       
 [31] tools_3.6.1              gtable_0.3.0             glue_1.3.1              
 [34] GenomeInfoDbData_1.2.2   reshape2_1.4.4           dplyr_0.8.5             
 [37] rappdirs_0.3.1           Rcpp_1.0.4               vctrs_0.3.1             
 [40] Biostrings_2.54.0        gdata_2.18.0             rtracklayer_1.46.0      
 [43] iterators_1.0.10         xfun_0.12                stringr_1.4.0           
 [46] lifecycle_0.2.0          gtools_3.8.1             XML_3.99-0.3            
 [49] dendextend_1.13.4        MASS_7.3-51.5            zlibbioc_1.32.0         
 [52] scales_1.1.0             TSP_1.1-10               pcaMethods_1.78.0       
 [55] hms_0.5.3                RColorBrewer_1.1-2       BBmisc_1.11             
 [58] curl_4.3                 memoise_1.1.0            heatmaply_1.1.0         
 [61] gridExtra_2.3            ggplot2_3.3.0            biomaRt_2.42.1          
 [64] rpart_4.1-15             latticeExtra_0.6-29      stringi_1.4.3           
 [67] RSQLite_2.2.0            genefilter_1.68.0        gclus_1.3.2             
 [70] foreach_1.4.4            checkmate_2.0.0          seriation_1.2-8         
 [73] caTools_1.17.1.2         rlang_0.4.6              pkgconfig_2.0.3         
 [76] bitops_1.0-6             lattice_0.20-38          purrr_0.3.3             
 [79] GenomicAlignments_1.22.1 htmlwidgets_1.5.1        bit_1.1-15.2            
 [82] tidyselect_1.1.0         plyr_1.8.6               magrittr_1.5            
 [85] R6_2.4.0                 gplots_3.0.1.1           Hmisc_4.4-0             
 [88] DBI_1.1.0                pillar_1.4.3             foreign_0.8-76          
 [91] survival_3.1-12          RCurl_1.98-1.1           nnet_7.3-13             
 [94] tibble_3.0.0             crayon_1.3.4             KernSmooth_2.23-15      
 [97] BiocFileCache_1.10.2     plotly_4.9.2.1           viridis_0.5.1           
[100] jpeg_0.1-8.1             progress_1.2.2           locfit_1.5-9.4          
[103] grid_3.6.1               blob_1.2.1               digest_0.6.18           
[106] webshot_0.5.2            xtable_1.8-4             tidyr_1.1.0             
[109] openssl_1.4.1            munsell_0.5.0            registry_0.5-1          
[112] viridisLite_0.3.0        askpass_1.1             

[c-mertes]

hey @emariella,

After looking into it, I'm not sure what is happening here. There are two parallelization mechanism that could trigger this.

1. The bplapply call. But this is set by you and also appropriately passed on to internal functions.
2. The intrinsic parallelized computation of matrix multiplications with BLAS and openMP that already caused some headack (Always hangs during autoencoding #18).

After reviewing the code again, the first option should not trigger it.

As we disabled in commit 1b9e5b5 openMP also this should not happen. To make sure, I explicitly set BLAS to use just one core in a new branch here f42b49c. As you use already 1.4.2 this should not happen, but maybe something went wrong with your Makevariables/CPPFlags while compiling the code.

Can you please try using the new branch version if this fixes the problem for you?

devtools::install_github("gagneurlab/OUTRIDER", ref="fix26")

# should give you the new line with `blas_set_num_threads(1L)`
OUTRIDER::findEncodingDim

# test it again
library(OUTRIDER)
ods <- makeExampleOutriderDataSet()

ncores <- 5
register(MulticoreParam(ncores, ncores*2, progressbar = TRUE))
ods <- findEncodingDim(ods, BPPARAM=bpparam(), params = seq(2,20,by=2))