Search
Question: Getting data/objects to workers in bplapply inside a function
1
gravatar for Pekka Kohonen
11 months ago by
Pekka Kohonen190
Sweden
Pekka Kohonen190 wrote:

I know that this question has probably been answered implicitly somewhere but I have read many answers online and it always seems to have been written in a way that I don't understand.

The situation is that I have a function which uses bplapply and I want to pass data from the function call to bplapply workers. What I have been doing so far (and this works to some degree) is taking all  the objects that I want the workers to have and putting them into a temporary file with a defined file name and loading that file inside the function that is executed by bplapply. But this is not very elegant and loading files is slow, even with SSDs.

My example:

analyze.some.data <- function(data.list, param1, param2, nproc) {

    require(BiocParallel)
    param <- SnowParam(workers = nproc, type = "SOCK")
    save(data.list, param1, param2, file="temp_file.RData")

     fun1 <- function(x) {
          require(limma)
          load(file="temp_file.RData")
          out <- romer(data.list[[x]], param1, param2, nrot=9999)
         return(out)
     }
     res <- bplapply(seq_along(data.list), fun1, BPPARAM = param)
     return(res)
}

And then you call it by:

final.result <- analyze.some.data(data.list=eSet.list, param1=genesets, param2=something, nproc=6)

So how do you do this without using the temporary file? Do you have additional variables in the fun1? But how will bplapply treat these since it is inputting already to one variable?

I used to use bplapply just like lapply but with lapply you still have access to the environment of the function (as I understand it), so there is no issue passing data to whatever you have inside the function executed by lapply.

ADD COMMENTlink modified 11 months ago by Martin Morgan ♦♦ 22k • written 11 months ago by Pekka Kohonen190
2
gravatar for Martin Morgan
11 months ago by
Martin Morgan ♦♦ 22k
United States
Martin Morgan ♦♦ 22k wrote:

The FUN of lapply can take additional arguments, passed as additional arguments to lapply, as

lapply(
    c("hi", "bye"),
    function(say, who)
        paste(say, who, Sys.getpid()),
    "Martin"
)

which work with bplapply

> lapply(c("hi", "bye"), function(say, who) paste(say, who, Sys.getpid()), "Martin")
[[1]]
[1] "hi Martin 7221"

[[2]]
[1] "bye Martin 7221"

> bplapply(c("hi", "bye"), function(say, who) paste(say, who, Sys.getpid()), "Martin")
[[1]]
[1] "hi Martin 7252"

[[2]]
[1] "bye Martin 7253"

The arguments can be named, and FUN follows the R rules for matching first named arguments and then unnamed arguments.

ADD COMMENTlink modified 11 months ago • written 11 months ago by Martin Morgan ♦♦ 22k

Thanks! It worked beautifully! A very clever example code.

ADD REPLYlink written 11 months ago by Pekka Kohonen190
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.2.0
Traffic: 282 users visited in the last hour