Search
Question: Getting data/objects to workers in bplapply inside a function
1
gravatar for Pekka Kohonen
6 weeks ago by
Pekka Kohonen190
Sweden
Pekka Kohonen190 wrote:

I know that this question has probably been answered implicitly somewhere but I have read many answers online and it always seems to have been written in a way that I don't understand.

The situation is that I have a function which uses bplapply and I want to pass data from the function call to bplapply workers. What I have been doing so far (and this works to some degree) is taking all  the objects that I want the workers to have and putting them into a temporary file with a defined file name and loading that file inside the function that is executed by bplapply. But this is not very elegant and loading files is slow, even with SSDs.

My example:

analyze.some.data <- function(data.list, param1, param2, nproc) {

    require(BiocParallel)
    param <- SnowParam(workers = nproc, type = "SOCK")
    save(data.list, param1, param2, file="temp_file.RData")

     fun1 <- function(x) {
          require(limma)
          load(file="temp_file.RData")
          out <- romer(data.list[[x]], param1, param2, nrot=9999)
         return(out)
     }
     res <- bplapply(seq_along(data.list), fun1, BPPARAM = param)
     return(res)
}

And then you call it by:

final.result <- analyze.some.data(data.list=eSet.list, param1=genesets, param2=something, nproc=6)

So how do you do this without using the temporary file? Do you have additional variables in the fun1? But how will bplapply treat these since it is inputting already to one variable?

I used to use bplapply just like lapply but with lapply you still have access to the environment of the function (as I understand it), so there is no issue passing data to whatever you have inside the function executed by lapply.

ADD COMMENTlink modified 6 weeks ago by Martin Morgan ♦♦ 20k • written 6 weeks ago by Pekka Kohonen190
2
gravatar for Martin Morgan
6 weeks ago by
Martin Morgan ♦♦ 20k
United States
Martin Morgan ♦♦ 20k wrote:

The FUN of lapply can take additional arguments, passed as additional arguments to lapply, as

lapply(
    c("hi", "bye"),
    function(say, who)
        paste(say, who, Sys.getpid()),
    "Martin"
)

which work with bplapply

> lapply(c("hi", "bye"), function(say, who) paste(say, who, Sys.getpid()), "Martin")
[[1]]
[1] "hi Martin 7221"

[[2]]
[1] "bye Martin 7221"

> bplapply(c("hi", "bye"), function(say, who) paste(say, who, Sys.getpid()), "Martin")
[[1]]
[1] "hi Martin 7252"

[[2]]
[1] "bye Martin 7253"

The arguments can be named, and FUN follows the R rules for matching first named arguments and then unnamed arguments.

ADD COMMENTlink modified 6 weeks ago • written 6 weeks ago by Martin Morgan ♦♦ 20k

Thanks! It worked beautifully! A very clever example code.

ADD REPLYlink written 6 weeks ago by Pekka Kohonen190
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.2.0
Traffic: 143 users visited in the last hour