Search
Question: bplapply() fails when BPPARAM = SnowParam() is specified
0
gravatar for jessica.hekman
3.9 years ago by
United States
jessica.hekman40 wrote:

I encountered this odd problem while debugging an issue with derfinder (see the original thread here: derfinder: analyzeChr() breaks on chromosomes > 22. This code causes my machine to hang and never return the R prompt:

 

library(BiocParallel)

xx <- bplapply(seq_len(1), function(x) sample(1:10, 1), BPPARAM = SnowParam())

However this code works as expected:

library(BiocParallel)

xx <- bplapply(seq_len(1), function(x) sample(1:10, 1), BPPARAM = MulticoreParam())

 

as does this:

library(BiocParallel)

xx <- bplapply(seq_len(1), function(x) sample(1:10, 1))

 

I've run that first snippet successfully on my Mac and on another Linux machine. The machine which won't run it is a 32 processor Fedora machine.

 

Thanks so much for any insights into what's going on!

> devtools::session_info()
Session info-------------------------------------------------------------------
 setting  value                       
 version  R version 3.1.1 (2014-07-10)
 system   x86_64, linux-gnu           
 ui       X11                         
 language (EN)                        
 collate  en_US.UTF-8                 
 tz       America/Chicago             

Packages-----------------------------------------------------------------------
 package      * version date       source        
 base64enc      0.1.2   2014-06-26 CRAN (R 3.1.1)
 BatchJobs      1.5     2014-10-30 CRAN (R 3.1.1)
 BBmisc         1.8     2014-10-30 CRAN (R 3.1.1)
 BiocGenerics   0.12.0  2014-10-15 Bioconductor  
 BiocParallel * 1.0.0   2014-11-11 Bioconductor  
 brew           1.0.6   2011-04-13 CRAN (R 3.1.1)
 checkmate      1.5.0   2014-10-19 CRAN (R 3.1.1)
 codetools      0.2.9   2014-08-21 CRAN (R 3.1.1)
 DBI            0.3.1   2014-09-24 CRAN (R 3.1.1)
 devtools       1.6.1   2014-10-07 CRAN (R 3.1.1)
 digest         0.6.4   2013-12-03 CRAN (R 3.0.2)
 fail           1.2     2013-09-19 CRAN (R 3.1.1)
 foreach        1.4.2   2014-04-11 CRAN (R 3.1.1)
 iterators      1.0.7   2014-04-11 CRAN (R 3.1.1)
 RSQLite        1.0.0   2014-10-25 CRAN (R 3.1.1)
 rstudioapi     0.1     2014-03-27 CRAN (R 3.1.1)
 sendmailR      1.2.1   2014-09-21 CRAN (R 3.1.1)
 stringr        0.6.2   2012-12-06 CRAN (R 3.0.2)

 

Jessica

 

 

ADD COMMENTlink written 3.9 years ago by jessica.hekman40
1

The cluster is being opened with

do.call(makeCluster, x$.clusterargs)

where makeCluster is from the parallel package (apparently the package is not included with devtools' sessionInfo(), which is too bad) and x$.clusterargs is SnowParam()$.clusterags. So for me I have

> SnowParam()$.clusterargs
$spec
[1] 8

$type
[1] "PSOCK"

and can create the cluster with

cl = parallel::makeCluster(spec=8, type="PSOCK")

It sounds like this will fail for you, and then yes, it's something about available ports on your system.  I believe you can create your SnowParam with the port argument, for instance, as documented on ?parallel::makeCluster.

If you're running this is as a parallel job on a single Linux / mac computer, then it's almost always better to use the (default) MulticoreParam.

ADD REPLYlink modified 3.9 years ago • written 3.9 years ago by Martin Morgan ♦♦ 22k

The parallel package is v3.1.1.

Correct, that line of code hangs on my machine. I'll talk with the admin and see if port access was changed recently in some way. I'll also ask the derfinder maintainer if he can use MulticoreParam instead of SnowParam.

 

Thank you!

Jessica

ADD REPLYlink written 3.9 years ago by jessica.hekman40

Regarding `devtools::session_info()` not printing the `base` package version, I just created an issue about this: https://github.com/hadley/devtools/issues/646

ADD REPLYlink written 3.9 years ago by Leonardo Collado Torres610

utils::sessionInfo() does not print base package versions either. Arguably it should.

ADD REPLYlink written 3.9 years ago by Dan Tenenbaum ♦♦ 8.2k

Martin edited his earlier comment and wanted the version of the `parallel` package. However, see https://github.com/hadley/devtools/issues/646#issuecomment-62804289 Like I said in https://github.com/hadley/devtools/issues/646#issuecomment-62837880 I can't use `install.packages('parallel')` on R 3.1.2 or 3.2, so maybe this was an issue in the past but it's no longer one. 

If you have a case where the versions of R and `parallel` could be un-linked, please share it. Otherwise I'll just close the github issue.

Thanks!

ADD REPLYlink written 3.9 years ago by Leonardo Collado Torres610

What is the output of just `SnowParam()`?

 

ADD REPLYlink written 3.9 years ago by Laurent Gatto1.0k
> SnowParam()
class: SnowParam; bpisup: FALSE; bpworkers: 8; catch.errors: TRUE
cluster spec: 8; type: PSOCK

ADD REPLYlink written 3.9 years ago by jessica.hekman40
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.2.0
Traffic: 198 users visited in the last hour