Question: Getting data/objects to workers in bplapply inside a function
1
20 months ago by
Pekka Kohonen190
Sweden
Pekka Kohonen190 wrote:

I know that this question has probably been answered implicitly somewhere but I have read many answers online and it always seems to have been written in a way that I don't understand.

The situation is that I have a function which uses bplapply and I want to pass data from the function call to bplapply workers. What I have been doing so far (and this works to some degree) is taking all  the objects that I want the workers to have and putting them into a temporary file with a defined file name and loading that file inside the function that is executed by bplapply. But this is not very elegant and loading files is slow, even with SSDs.

My example:

analyze.some.data <- function(data.list, param1, param2, nproc) {

require(BiocParallel)
param <- SnowParam(workers = nproc, type = "SOCK")
save(data.list, param1, param2, file="temp_file.RData")

fun1 <- function(x) {
require(limma)
out <- romer(data.list[[x]], param1, param2, nrot=9999)
return(out)
}
res <- bplapply(seq_along(data.list), fun1, BPPARAM = param)
return(res)
}

And then you call it by:

final.result <- analyze.some.data(data.list=eSet.list, param1=genesets, param2=something, nproc=6)

So how do you do this without using the temporary file? Do you have additional variables in the fun1? But how will bplapply treat these since it is inputting already to one variable?

I used to use bplapply just like lapply but with lapply you still have access to the environment of the function (as I understand it), so there is no issue passing data to whatever you have inside the function executed by lapply.

biocparallel • 358 views
modified 20 months ago by Martin Morgan ♦♦ 23k • written 20 months ago by Pekka Kohonen190
Answer: Getting data/objects to workers in bplapply inside a function
2
20 months ago by
Martin Morgan ♦♦ 23k
United States
Martin Morgan ♦♦ 23k wrote:

The FUN of lapply can take additional arguments, passed as additional arguments to lapply, as

lapply(
c("hi", "bye"),
function(say, who)
paste(say, who, Sys.getpid()),
"Martin"
)

which work with bplapply

> lapply(c("hi", "bye"), function(say, who) paste(say, who, Sys.getpid()), "Martin")
[[1]]
[1] "hi Martin 7221"

[[2]]
[1] "bye Martin 7221"

> bplapply(c("hi", "bye"), function(say, who) paste(say, who, Sys.getpid()), "Martin")
[[1]]
[1] "hi Martin 7252"

[[2]]
[1] "bye Martin 7253"

The arguments can be named, and FUN follows the R rules for matching first named arguments and then unnamed arguments.