Create multiple DoparParams with different number of registered cores
2
0
Entering edit mode
ATpoint ★ 4.2k
@atpoint-13662
Last seen 1 hour ago
Germany

My question is actually simple: How can I manage to create several DoparParam() objects, each with a different number of registered cores? See below for what I tried. In a nutshell, it seems I can only have one DoparParam() object, because once I run registerDoParallel() existing objects will be overwritten in terms of number of registred cores.

Why is that important? Say I need one DoparParam with just very few cores, because the bplapply will be very memory-intensive, and one with many cores, because the bplapply will not be memory-intensive at all. How can I do that? Please do not suggest to use any of the other available backends. DoparParam (in my hands) works the smoothest, on all OS, with least problems, so I really would like to use that, but in a more flexible fashion in terms of having multiple ones available with different numbers of cores, and without running registerDoParallel before each bplapply, but rather once in a setup script, and then go along with the objects all the way through the project.

suppressMessages({

  library(BiocParallel)
  library(doParallel)

})

registerDoParallel(cores = 2)
bp2 <- DoparParam()
bp2
#> class: DoparParam
#>   bpisup: TRUE; bpnworkers: 2; bptasks: 0; bpjobname: BPJOB
#>   bplog: FALSE; bpthreshold: INFO; bpstopOnError: TRUE
#>   bpRNGseed: ; bptimeout: NA; bpprogressbar: FALSE
#>   bpexportglobals: TRUE; bpexportvariables: TRUE; bpforceGC: FALSE
#>   bpfallback: TRUE
#>   bplogdir: NA
#>   bpresultdir: NA

# no effect, still using 2 cores as workers
bp2$workers <- 18L
bp2
#> class: DoparParam
#>   bpisup: TRUE; bpnworkers: 2; bptasks: 0; bpjobname: BPJOB
#>   bplog: FALSE; bpthreshold: INFO; bpstopOnError: TRUE
#>   bpRNGseed: ; bptimeout: NA; bpprogressbar: FALSE
#>   bpexportglobals: TRUE; bpexportvariables: TRUE; bpforceGC: FALSE
#>   bpfallback: TRUE
#>   bplogdir: NA
#>   bpresultdir: NA

# no effect, still using 2 cores as workers
bptasks(bp2) <- 18L
bp2
#> class: DoparParam
#>   bpisup: TRUE; bpnworkers: 2; bptasks: 18; bpjobname: BPJOB
#>   bplog: FALSE; bpthreshold: INFO; bpstopOnError: TRUE
#>   bpRNGseed: ; bptimeout: NA; bpprogressbar: FALSE
#>   bpexportglobals: TRUE; bpexportvariables: TRUE; bpforceGC: FALSE
#>   bpfallback: TRUE
#>   bplogdir: NA
#>   bpresultdir: NA

# only valid for some Snow and MultiCore
bpworkers(bp2) <- 18L
#> Error in (function (classes, fdef, mtable) : unable to find an inherited method for function 'bpworkers<-' for signature '"DoparParam", "integer"'

# overwrites bp2
registerDoParallel(cores = 18)
bp18 <- DoparParam()
bp18
#> class: DoparParam
#>   bpisup: TRUE; bpnworkers: 18; bptasks: 0; bpjobname: BPJOB
#>   bplog: FALSE; bpthreshold: INFO; bpstopOnError: TRUE
#>   bpRNGseed: ; bptimeout: NA; bpprogressbar: FALSE
#>   bpexportglobals: TRUE; bpexportvariables: TRUE; bpforceGC: FALSE
#>   bpfallback: TRUE
#>   bplogdir: NA
#>   bpresultdir: NA
bp2
#> class: DoparParam
#>   bpisup: TRUE; bpnworkers: 18; bptasks: 18; bpjobname: BPJOB
#>   bplog: FALSE; bpthreshold: INFO; bpstopOnError: TRUE
#>   bpRNGseed: ; bptimeout: NA; bpprogressbar: FALSE
#>   bpexportglobals: TRUE; bpexportvariables: TRUE; bpforceGC: FALSE
#>   bpfallback: TRUE
#>   bplogdir: NA
#>   bpresultdir: NA

sessionInfo()
#> R version 4.3.2 (2023-10-31)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 22.04.3 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] parallel  stats     graphics  grDevices utils     datasets  methods  
#> [8] base     
#> 
#> other attached packages:
#> [1] doParallel_1.0.17   iterators_1.0.14    foreach_1.5.2      
#> [4] BiocParallel_1.36.0
#> 
#> loaded via a namespace (and not attached):
#>  [1] digest_0.6.33     codetools_0.2-19  fastmap_1.1.1     xfun_0.41        
#>  [5] glue_1.6.2        knitr_1.45        htmltools_0.5.7   rmarkdown_2.25   
#>  [9] lifecycle_1.0.4   cli_3.6.1         reprex_2.0.2      withr_2.5.2      
#> [13] compiler_4.3.2    rstudioapi_0.15.0 tools_4.3.2       evaluate_0.23    
#> [17] yaml_2.3.7        rlang_1.1.2       fs_1.6.3
Created on 2024-06-17 with reprex v2.0.2
BiocParallel • 531 views
ADD COMMENT
2
Entering edit mode
@martin-morgan-1513
Last seen 8 days ago
United States

I think the answer is "you can't". DoparParam() provides a 'wrapper' around doParallel functionality, e.g., bpworkers(DoparParam()) reads foreach::getDoParWorkers(), which references the most recently registered cluster.

ADD COMMENT
0
Entering edit mode
@james-w-macdonald-5106
Last seen 15 hours ago
United States

The doParallel package is a CRAN package, not Bioconductor, so you could ask over on r-help@r-project.org, or maybe biostars.org or stackoverflow.com

You can register any number of parallel backends using BiocParallel though. See the vignette.

0
Entering edit mode

This does not answer my question. I am not aiming to register different types of backends, but the same backend with different configurations. BiocParallel is obviously a Bioc package, so it's on topic.

ADD REPLY
0
Entering edit mode

registerDoParallel and DoparParam are both doParallel functions, which again, is a CRAN package. What part of the code you present above is meant to be used within BiocParallel?

0
Entering edit mode

https://github.com/Bioconductor/BiocParallel/blob/devel/R/DoparParam-class.R

DoparParam is a BiocParallel class, that is why I am asking here.

ADD REPLY
0
Entering edit mode

OK, my bad. Never used that aspect of the package before.

Login before adding your answer.

Traffic: 814 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6