bplapply() fails when BPPARAM = SnowParam() is specified
0
0
Entering edit mode
@jessicahekman-6877
Last seen 6.1 years ago
United States

I encountered this odd problem while debugging an issue with derfinder (see the original thread here: derfinder: analyzeChr() breaks on chromosomes > 22. This code causes my machine to hang and never return the R prompt:

 

library(BiocParallel)

xx <- bplapply(seq_len(1), function(x) sample(1:10, 1), BPPARAM = SnowParam())

However this code works as expected:

library(BiocParallel)

xx <- bplapply(seq_len(1), function(x) sample(1:10, 1), BPPARAM = MulticoreParam())

 

as does this:

library(BiocParallel)

xx <- bplapply(seq_len(1), function(x) sample(1:10, 1))

 

I've run that first snippet successfully on my Mac and on another Linux machine. The machine which won't run it is a 32 processor Fedora machine.

 

Thanks so much for any insights into what's going on!

> devtools::session_info()
Session info-------------------------------------------------------------------
 setting  value                       
 version  R version 3.1.1 (2014-07-10)
 system   x86_64, linux-gnu           
 ui       X11                         
 language (EN)                        
 collate  en_US.UTF-8                 
 tz       America/Chicago             

Packages-----------------------------------------------------------------------
 package      * version date       source        
 base64enc      0.1.2   2014-06-26 CRAN (R 3.1.1)
 BatchJobs      1.5     2014-10-30 CRAN (R 3.1.1)
 BBmisc         1.8     2014-10-30 CRAN (R 3.1.1)
 BiocGenerics   0.12.0  2014-10-15 Bioconductor  
 BiocParallel * 1.0.0   2014-11-11 Bioconductor  
 brew           1.0.6   2011-04-13 CRAN (R 3.1.1)
 checkmate      1.5.0   2014-10-19 CRAN (R 3.1.1)
 codetools      0.2.9   2014-08-21 CRAN (R 3.1.1)
 DBI            0.3.1   2014-09-24 CRAN (R 3.1.1)
 devtools       1.6.1   2014-10-07 CRAN (R 3.1.1)
 digest         0.6.4   2013-12-03 CRAN (R 3.0.2)
 fail           1.2     2013-09-19 CRAN (R 3.1.1)
 foreach        1.4.2   2014-04-11 CRAN (R 3.1.1)
 iterators      1.0.7   2014-04-11 CRAN (R 3.1.1)
 RSQLite        1.0.0   2014-10-25 CRAN (R 3.1.1)
 rstudioapi     0.1     2014-03-27 CRAN (R 3.1.1)
 sendmailR      1.2.1   2014-09-21 CRAN (R 3.1.1)
 stringr        0.6.2   2012-12-06 CRAN (R 3.0.2)

 

Jessica

 

 

biocparallel • 1.6k views
ADD COMMENT
1
Entering edit mode

The cluster is being opened with

do.call(makeCluster, x$.clusterargs)

where makeCluster is from the parallel package (apparently the package is not included with devtools' sessionInfo(), which is too bad) and x$.clusterargs is SnowParam()$.clusterags. So for me I have

> SnowParam()$.clusterargs
$spec
[1] 8

$type
[1] "PSOCK"

and can create the cluster with

cl = parallel::makeCluster(spec=8, type="PSOCK")

It sounds like this will fail for you, and then yes, it's something about available ports on your system.  I believe you can create your SnowParam with the port argument, for instance, as documented on ?parallel::makeCluster.

If you're running this is as a parallel job on a single Linux / mac computer, then it's almost always better to use the (default) MulticoreParam.

ADD REPLY
0
Entering edit mode

The parallel package is v3.1.1.

Correct, that line of code hangs on my machine. I'll talk with the admin and see if port access was changed recently in some way. I'll also ask the derfinder maintainer if he can use MulticoreParam instead of SnowParam.

 

Thank you!

Jessica

ADD REPLY
0
Entering edit mode

Regarding `devtools::session_info()` not printing the `base` package version, I just created an issue about this: https://github.com/hadley/devtools/issues/646

ADD REPLY
0
Entering edit mode

utils::sessionInfo() does not print base package versions either. Arguably it should.

ADD REPLY
0
Entering edit mode

Martin edited his earlier comment and wanted the version of the `parallel` package. However, see https://github.com/hadley/devtools/issues/646#issuecomment-62804289 Like I said in https://github.com/hadley/devtools/issues/646#issuecomment-62837880 I can't use `install.packages('parallel')` on R 3.1.2 or 3.2, so maybe this was an issue in the past but it's no longer one. 

If you have a case where the versions of R and `parallel` could be un-linked, please share it. Otherwise I'll just close the github issue.

Thanks!

ADD REPLY
0
Entering edit mode

What is the output of just `SnowParam()`?

 

ADD REPLY
0
Entering edit mode
> SnowParam()
class: SnowParam; bpisup: FALSE; bpworkers: 8; catch.errors: TRUE
cluster spec: 8; type: PSOCK

ADD REPLY

Login before adding your answer.

Traffic: 241 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6