Tutorial:doParallel foreach already exporting variable(s)
0
0
Entering edit mode
solo7773 ▴ 10
@solo7773-11037
Last seen 7.2 years ago

According to my use of doParallel, I share my experience here.

refGenome = BSgenome.Hsapiens.UCSC.hg19::BSgenome.Hsapiens.UCSC.hg19
# start parallel
  library(doParallel)
  nCPUcores = detectCores()
  if (nCPUcores < 3) {
    registerDoSEQ()
  }else{
    cl = makeCluster(nCPUcores - 1)
    registerDoParallel(cl)
  }

 

# example foreach 1
resParallel = foreach(g = 1:1000) %dopar% {
  # code omitted
}

 

# example foreach 2
resParallel = foreach(g = 1:1000, .export = c('refGenome')) %dopar% {
​  # code omitted
}

# example foreach 3
resParallel = foreach(g = 1:1000, .packages = c('BSgenome.Hsapiens.UCSC.hg19')) %dopar% {
  # code omitted
}

 

Explanation:

1) example foreach 1 throws fatal error, something like 'task 1 failed', 'object *** not found'

 

2) example foreach 2 works but gives warning like 'already exporting variable(s): refGenome'

 

Case 1) and 2) usually happen on Windows, because foreach use its own environment. When use functions or packages from outside, you need to use .export or .packages to make them available inside the foreach loop.

 

3) example foreach 3 finally does my work

 

I have to acknowledge that lots of solutions for case 1) are available online and I also benefit and learned from them. They help me to find ways for case 2). I suppress the annoying warning message. Hope it'll also be useful for you guys.

doParallel foreach export Tutorial • 7.4k views
ADD COMMENT

Login before adding your answer.

Traffic: 783 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6