Question: bplapply 1.2.9 (and higher) breaks on Windows x64
0
4.4 years ago by
genfys0
Sweden
genfys0 wrote:

Hi,

There seems to be a problem with bplapply() on Windows in BiocParallel for version 1.2.9 and higher (I tested up to 1.3.41). While the code below works fine on BiocParallel version 1.0.3 it causes the system to hang on newer versions. Can anybody reproduce this problem or else tell me what I got wrong...

Minimal test code:

library(parallel)
cl<-makeCluster(2, 'SOCK')
parLapply(cl, 1:4, function(i) Sys.sleep(1))
stopCluster(cl)

library(BiocParallel)
register(SnowParam(workers = 2, type = 'SOCK'), default = TRUE)
bpparam()
bplapply(1:4, function(i) Sys.sleep(1))

While the first snippet works fine on Linux, OSX and Windows, the second one hangs up on Windows. When stopping execution, I get the following warning:

Warning message:
running command 'env MASTER=localhost PORT=11521 OUT=/dev/null RPROG=C:/PROGRA~1/R/R-32~1.1/bin/R R_LIBS= C:/Users/loge/Documents/R/win-library/3.2/BiocParallel/RSOCKnode.sh' had status 127

Cheers

Lorenz

biocparallel • 931 views
modified 4.4 years ago • written 4.4 years ago by genfys0

I wonder if RPROG=C:/PROGRA~1/R/R-32~1.1/bin/R points to the 32 or 64-bit version of the application? Also, the output of sessionInfo() is always helpful...

Morgan, the above mentioned path leads to a local R library directory. On my current setup, there is just the x64 version installed. Find below the session info:

> sessionInfo() R version 3.2.1 (2015-06-18) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 7 x64 (build 7601) Service Pack 1

locale: [1] LC_COLLATE=Swedish_Sweden.1252  LC_CTYPE=Swedish_Sweden.1252    LC_MONETARY=Swedish_Sweden.1252 [4] LC_NUMERIC=C                    LC_TIME=Swedish_Sweden.1252    

attached base packages: [1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages: [1] BiocParallel_1.3.41

loaded via a namespace (and not attached): [1] parallel_3.2.1       futile.logger_1.4.1  tools_3.2.1          lambda.r_1.1.7       [5] futile.options_1.0.0 > 

Answer: bplapply 1.2.9 (and higher) breaks on Windows x64
1
4.4 years ago by
genfys0
Sweden
genfys0 wrote:

When I install cygwin and put it on the path everything works.

Lorenz

Just to clarify, BiocParallel currently uses useRscript=FALSE so it can point to a different worker script - one inside BiocParallel instead of snow or parallel. Initially this path seemed like the most direct but light-handed approach, the one requiring the least code duplication, etc. The shell script is clearly a problem on windows machines without a shell emulator so I'm working to fix this.

As an fyi, the script in BiocParallel adds these features on top of a cluster from snow or parallel:

* logging that captures runtime, memory use and futile.logger log messages

* ability to write out result and log files vs just return to workspace

* ability to set timeout on workers (max time per task)

* error handling / catching

Valerie

I've checked in a fix to BiocParallel 1.3.42. Workers are now launched with an R script by default vs shell. This approach was more straightforward than expected and in hindsight is probably the one I should have taken first.

I'm still doing some testing but wanted to give you a heads up if you wanted to try it. Now that you are cygwin enabled it may not matter much but at least a proper fix is out there.

Thanks for reporting this behavior.

Valerie

Ok, great, thanks. I have another small thing loosely related to the topic. Currently when I load BiocParallel on a 2 core machine, it ends up with 0 workers which breaks my code if I don't do some additional checking. Wouldn't it make sense to register directly SerialParam when there are less than =< 2 cores?

Yes, good idea. Now done in 1.2.16 and 1.3.44.

Valerie

Answer: bplapply 1.2.9 (and higher) breaks on Windows x64
0
4.4 years ago by
United States
Valerie Obenchain6.7k wrote:

Hi Lorenz,

I can't reproduce this error but that doesn't mean there isn't a problem. BiocParallel always uses a shell script (inst/RSOCKnode.sh) to launch the workers whereas snow launches via an R script or shell script (default is R script). The cluster arg that controls this behavior is useRscript.

Can you try running both a shell and R script created cluster on your machine? This may at least identify if the problem is due to the system call -

library(snow) cl1 <- makeCluster(2, "SOCK", useRscript=TRUE, outfile=NULL) cl2 <- makeCluster(2, "SOCK", useRscript=FALSE, outfile=NULL) parLapply(cl1, 1:2, Sys.sleep) parLapply(cl2, 1:2, Sys.sleep)

Here is my output:

> cl1 <- makeCluster(2, "SOCK", useRscript=TRUE, outfile=NULL)
starting worker for localhost:11270
starting worker for localhost:11270
> cl2 <- makeCluster(2, "SOCK", useRscript=FALSE, outfile=NULL)
> parLapply(cl1, 1:2, Sys.sleep)
Type: EXEC
Type: EXEC
[[1]]
NULL

[[2]]
NULL

> parLapply(cl2, 1:2, Sys.sleep)
[[1]]
NULL

[[2]]
NULL

Valerie

Hi Valerie,

I get the hang up already on:

> cl2 <- makeCluster(2, "SOCK", useRscript=FALSE, outfile=NULL)

So I assume it's not a BiocParallel problem, but rather that I have some wrong setup. Do I need to have something like cygwin installed for RSOCKnode.sh to run (sorry, I'm not used to windows)? I have RTools33.exe installed but not a full cygwin.

It looks like a problem in snow on both 32 and 64 bit. We have RTools on our build machine which is where I did my testing and could not reproduce the problem. I would have thought having RTools installed would work for you. Installing full cygwin may fix the problem but it should be viewed as a workaround. We are working on a more permanent solution.

Valerie