BiocParallel not proceeding
1
1
Entering edit mode
petyuk ▴ 70
@petyuk-7261
Last seen 20 months ago
United States

After refreshing my R installation and the packages, this simple example of BiocParallel::bplapply doesn't work properly.  It looks like it starts, but never finishes.

Thanks for taking a look into it!

Vlad

 

> library(BiocParallel)
> fun <- function(v) {
+     message("working") ## 10 tasks
+     sqrt(v)
+ }
> bplapply(1:10, fun)

# it does not finish (at least in a minute or so)

> sessionInfo()
R version 3.3.1 (2016-06-21)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.10.4 (Yosemite)

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] BiocParallel_1.6.6

loaded via a namespace (and not attached):
[1] parallel_3.3.1 tools_3.3.1   
> 
software error • 1.3k views
ADD COMMENT
0
Entering edit mode
  1. upgrading to BiocParallel_1.7.9 didn't help
  2. works fine on windows machine though

 

ADD REPLY
0
Entering edit mode

It hits a snag at this line in bplapply

BPPARAM <- bpstart(BPPARAM, length(X))

ADD REPLY
0
Entering edit mode

Within bpstart the issue is at

bpbackend(x) <- .bpmakeForkCluster(nnodes, bptimeout(x), getOption("ports", NA_integer_))

ADD REPLY
0
Entering edit mode

A likely candidate is that ports on the Mac are not accessible; I'm not familiar enough with how things work, but I think the overall strategy would be to identify open ports and use these via options(ports=123).

ADD REPLY
0
Entering edit mode

Not sure if this is relevant but this is what I get (regardless of the port). Although may be this is what is supposed to be since there is no process communicating to the 11709 port.  Anyway, BiocParallel:::.bpmakeForkCluster is not cooperating with me today.

> port <- 11000L + sample(1000L, 1L)
> port
[1] 11709
> socketConnection("localhost", port=port, TRUE, TRUE, "a+b", timeout = 3)
Error in socketConnection("localhost", port = port, TRUE, TRUE, "a+b",  : 
  cannot open the connection
In addition: Warning message:
In socketConnection("localhost", port = port, TRUE, TRUE, "a+b",  :
  problem in listening on this socket
ADD REPLY
0
Entering edit mode

What is odd it starts with MulticoreParam(), that is FORK type, but later switches to SOCK type.  I thought SOCK is for Windows, but not POSIX systems.

ADD REPLY
0
Entering edit mode

The 'SOCK' class of the return value is a red herring here. A simple fork isn't rich enough to support the BiocParallel behavior, which requires communication between the forked and master process. Sockets are used for that communication.

ADD REPLY
0
Entering edit mode

 

If I intercept the execution and rename the host from "WE25743" (the name of my machine) to "localhost", then .bpmakeForkCluster proceeds just fine. Hope that helps.

 

debugging in: BiocParallel:::.bpmakeForkCluster(8, 10, NA_integer_)
debug: {
    nnodes <- as.integer(nnodes)
    if (is.na(nnodes) || nnodes < 1L)
        stop("'nnodes' must be >= 1")
    if (is.na(port))
        port <- 11000L + sample(1000L, 1L)
    else if (length(port) != 1L)
        stop("'port' must be integer(1)")
    host <- Sys.info()[["nodename"]]
    cl <- vector("list", nnodes)
    for (rank in seq_len(nnodes)) {
        .bpmakeForkChild(host, port, rank, timeout)
        cl[[rank]] <- .bpconnectForkChild(host, port, rank, timeout)
    }
    class(cl) <- c("SOCKcluster", "cluster")
    cl
}

 

ADD REPLY
1
Entering edit mode
@martin-morgan-1513
Last seen 11 hours ago
United States

Sorry for the delay and thanks for debugging. I can sort of guess at the issue and have provided a patched version in BiocParallel 1.8.1 (current Bioc release version 3.4) and 1.9.1 (bioc devel), available with either

BiocInstaller::biocLite("Bioconductor-mirror/BiocParallel@release-3.4")  ## if you're using Bioconductor v. 3.4, release
BiocInstaller::biocLite("Bioconductor-mirror/BiocParallel")  ## if you're using Bioconductor v. 3.5, devel

or after the next build (likely Sunday afternoon, eastern time) via biocLite("BiocParallel").

The workaround lets you set the host using a global option

options(bphost="localhost")

instead of the  default Sys.info()[["nodename"]]. Set the option any time before calling bplapply(), bpvec(), or bpstart(). I'd  appreciate hearing whether this works for you (and others), and if so will make a more permanent change.

ADD COMMENT

Login before adding your answer.

Traffic: 235 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6