Search
Question: doParallel with Bumphunter gives Error in mcfork()
0
gravatar for Guest User
3.6 years ago by
Guest User12k
Guest User12k wrote:
I'm trying to run bumphunter within the minfi package on a linux cluster, and when I use doParallel with bumphunter it consistently errors out with "Error in mcfork() : unable to fork, possible reason: Cannot allocate memory" I haven't been able to find a solution searching the web or working with our support staff. Has anyone seen this before or know where the problem is? Any help will be greatly appreciated. - Brad -- output of sessionInfo(): Loading module hpc/intel-mkl-11.0.0.079. JDK 1.6 module **************************************************** This module sets up the following environment variables for JDK 1.6: PATH /n/app/jdk1.6.0_11/bin **************************************************** Loading module centos6/tcl-8.5.14. Loading module centos6/tk-8.5.14. Loading module centos6/fftw-3.3_gcc-4.4.7. Loading module centos6/gsl-1.16_gcc-4.4.7. Loading module centos6/hdf5-1.8.11_gcc-4.4.7. Loading module centos6/netcdf-4.3.0_gcc-4.4.7. Loading module centos6/R-3.0.2. Loading required package: minfi Loading required package: methods Loading required package: BiocGenerics Loading required package: parallel Attaching package: ???????BiocGenerics???????? The following objects are masked from ???????package:parallel????????: clusterApply, clusterApplyLB, clusterCall, clusterEvalQ, clusterExport, clusterMap, parApply, parCapply, parLapply, parLapplyLB, parRapply, parSapply, parSapplyLB The following object is masked from ???????package:stats????????: xtabs The following objects are masked from ???????package:base????????: anyDuplicated, append, as.data.frame, as.vector, cbind, colnames, duplicated, eval, evalq, Filter, Find, get, intersect, is.unsorted, lapply, Map, mapply, match, mget, order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank, rbind, Reduce, rep.int, rownames, sapply, setdiff, sort, table, tapply, union, unique, unlist Loading required package: Biobase Welcome to Bioconductor Vignettes contain introductory material; view with 'browseVignettes()'. To cite Bioconductor, see 'citation("Biobase")', and for packages 'citation("pkgname")'. Loading required package: lattice Loading required package: reshape Loading required package: plyr Attaching package: ???????reshape???????? The following objects are masked from ???????package:plyr????????: rename, round_any Loading required package: GenomicRanges Loading required package: IRanges Attaching package: ???????IRanges???????? The following objects are masked from ???????package:reshape????????: expand, rename The following objects are masked from ???????package:plyr????????: desc, rename Loading required package: XVector Attaching package: ???????XVector???????? The following object is masked from ???????package:plyr????????: compact Loading required package: Biostrings Loading required package: bumphunter Loading required package: foreach Loading required package: iterators Loading required package: locfit locfit 1.5-9.1 2013-03-22 Attaching package: ???????locfit???????? The following objects are masked from ???????package:GenomicRanges????????: left, right Loading required package: IlluminaHumanMethylation450kmanifest Loading required package: IlluminaHumanMethylation450kanno.ilmn12.hg19 [bumphunterEngine] Parallelizing using 64 workers/cores (backend: doParallel, version: 1.0.6). [bumphunterEngine] Computing coefficients. [bumphunterEngine] Performing 1000 permutations. [bumphunterEngine] Computing marginal permutation p-values. [bumphunterEngine] cutoff: 0.05 [bumphunterEngine] Finding regions. [bumphunterEngine] Found 175279 bumps. [bumphunterEngine] Computing regions for each permutation. Loading required package: rngtools Loading required package: pkgmaker Loading required package: registry Attaching package: ???????pkgmaker???????? The following object is masked from ???????package:IRanges????????: new2 Error in mcfork() : unable to fork, possible reason: Cannot allocate memory Calls: bumphunter ... %dopar% -> <anonymous> -> mclapply -> lapply -> FUN -> mcfork Execution halted -- Sent via the guest posting facility at bioconductor.org.
ADD COMMENTlink modified 3.6 years ago by Tim Triche4.2k • written 3.6 years ago by Guest User12k
0
gravatar for Vincent J. Carey, Jr.
3.6 years ago by
United States
Vincent J. Carey, Jr.6.2k wrote:
On Sat, Apr 12, 2014 at 10:42 PM, Brad Ruzicka [guest] < guest@bioconductor.org> wrote: > > I'm trying to run bumphunter within the minfi package on a linux cluster, > and when I use doParallel with bumphunter it consistently errors out with > "Error in mcfork() : unable to fork, possible reason: Cannot allocate > memory" > I haven't been able to find a solution searching the web or working with > our support staff. > Has anyone seen this before or know where the problem is? > Any help will be greatly appreciated. > - Brad > > > -- output of sessionInfo(): > > Loading module hpc/intel-mkl-11.0.0.079. > > JDK 1.6 module > **************************************************** > > This module sets up the following environment > variables for JDK 1.6: > PATH /n/app/jdk1.6.0_11/bin > > **************************************************** > > Loading module centos6/tcl-8.5.14. > Loading module centos6/tk-8.5.14. > Loading module centos6/fftw-3.3_gcc-4.4.7. > Loading module centos6/gsl-1.16_gcc-4.4.7. > Loading module centos6/hdf5-1.8.11_gcc-4.4.7. > Loading module centos6/netcdf-4.3.0_gcc-4.4.7. > Loading module centos6/R-3.0.2. > Loading required package: minfi > Loading required package: methods > Loading required package: BiocGenerics > Loading required package: parallel > > Attaching package: ‘BiocGenerics’ > > The following objects are masked from ‘package:parallel’: > > clusterApply, clusterApplyLB, clusterCall, clusterEvalQ, > clusterExport, clusterMap, parApply, parCapply, parLapply, > parLapplyLB, parRapply, parSapply, parSapplyLB > > The following object is masked from ‘package:stats’: > > xtabs > > The following objects are masked from ‘package:base’: > > anyDuplicated, append, as.data.frame, as.vector, cbind, colnames, > duplicated, eval, evalq, Filter, Find, get, intersect, is.unsorted, > lapply, Map, mapply, match, mget, order, paste, pmax, pmax.int, > pmin, pmin.int, Position, rank, rbind, Reduce, rep.int, rownames, > sapply, setdiff, sort, table, tapply, union, unique, unlist > > Loading required package: Biobase > Welcome to Bioconductor > > Vignettes contain introductory material; view with > 'browseVignettes()'. To cite Bioconductor, see > 'citation("Biobase")', and for packages 'citation("pkgname")'. > > Loading required package: lattice > Loading required package: reshape > Loading required package: plyr > > Attaching package: ‘reshape’ > > The following objects are masked from ‘package:plyr’: > > rename, round_any > > Loading required package: GenomicRanges > Loading required package: IRanges > > Attaching package: ‘IRanges’ > > The following objects are masked from ‘package:reshape’: > > expand, rename > > The following objects are masked from ‘package:plyr’: > > desc, rename > > Loading required package: XVector > > Attaching package: ‘XVector’ > > The following object is masked from ‘package:plyr’: > > compact > > Loading required package: Biostrings > Loading required package: bumphunter > Loading required package: foreach > Loading required package: iterators > Loading required package: locfit > locfit 1.5-9.1 2013-03-22 > > Attaching package: ‘locfit’ > > The following objects are masked from ‘package:GenomicRanges’: > > left, right > > Loading required package: IlluminaHumanMethylation450kmanifest > Loading required package: IlluminaHumanMethylation450kanno.ilmn12.hg19 > [bumphunterEngine] Parallelizing using 64 workers/cores (backend: > doParallel, version: 1.0.6). > i don't think it is possible to say exactly how to get around this on the basis of this information. You have to match the request to the available resources. One theory: at some point you ran registerDoParallel(cores=64) to get the message above. The image that is forked has memory footprint F, say, and the machine that will run the job must have at least 64*F memory available to you to succeed with this request. If you reduce the request on the number of cores, you may complete the job; I don't know enough about the whole setup to be more precise. Another theory -- you have a cluster of 64 machines but at least one of them does not have enough RAM to satisfy whatever mcfork requests occured there. In principle you can shape the request to use multiple cores on multiple independent machines but you have to respect the limits of the various resources. Experimenting with a partial run to estimate what will be needed is a useful and often essential step. > [bumphunterEngine] Computing coefficients. > [bumphunterEngine] Performing 1000 permutations. > [bumphunterEngine] Computing marginal permutation p-values. > [bumphunterEngine] cutoff: 0.05 > [bumphunterEngine] Finding regions. > [bumphunterEngine] Found 175279 bumps. > [bumphunterEngine] Computing regions for each permutation. > Loading required package: rngtools > Loading required package: pkgmaker > Loading required package: registry > > Attaching package: ‘pkgmaker’ > > The following object is masked from ‘package:IRanges’: > > new2 > > Error in mcfork() : > unable to fork, possible reason: Cannot allocate memory > Calls: bumphunter ... %dopar% -> <anonymous> -> mclapply -> lapply -> FUN > -> mcfork > Execution halted > > > -- > Sent via the guest posting facility at bioconductor.org. > > _______________________________________________ > Bioconductor mailing list > Bioconductor@r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: > http://news.gmane.org/gmane.science.biology.informatics.conductor > [[alternative HTML version deleted]]
ADD COMMENTlink written 3.6 years ago by Vincent J. Carey, Jr.6.2k
0
gravatar for Tim Triche
3.6 years ago by
Tim Triche4.2k
United States
Tim Triche4.2k wrote:
Send your sessionInfo(). While you're doing that, here are some suggestions: 1) use fewer processes. minfi will kill a machine with 64GB of RAM using as few as 4 processes for bumphunting. Plus, you probably want to use doMC() instead of doParallel(). Try setting options("mc.cores") to something like 4 and gradually increasing it as you successfully run analyses on this machine/cluster/whatever. 2) look into using DMRcate if it's a simple 1:1 comparison. It's faster than bumphunter. 3) the real drag here is that it looks like your run actually completes!! You have a huge number of candidate bumps but when the permutation testing is being run, you run out of memory for at least one of the child processes, and that's all she wrote. So, back to #1, use fewer processes for the time being. If you can put more RAM into the machine, by all means do so, but until that is done, try using a relatively small number of processes. Best, --t Statistics is the grammar of science. Karl Pearson <http: en.wikipedia.org="" wiki="" the_grammar_of_science=""> On Sat, Apr 12, 2014 at 7:42 PM, Brad Ruzicka [guest] < guest@bioconductor.org> wrote: > > I'm trying to run bumphunter within the minfi package on a linux cluster, > and when I use doParallel with bumphunter it consistently errors out with > "Error in mcfork() : unable to fork, possible reason: Cannot allocate > memory" > I haven't been able to find a solution searching the web or working with > our support staff. > Has anyone seen this before or know where the problem is? > Any help will be greatly appreciated. > - Brad > > > -- output of sessionInfo(): > > Loading module hpc/intel-mkl-11.0.0.079. > > JDK 1.6 module > **************************************************** > > This module sets up the following environment > variables for JDK 1.6: > PATH /n/app/jdk1.6.0_11/bin > > **************************************************** > > Loading module centos6/tcl-8.5.14. > Loading module centos6/tk-8.5.14. > Loading module centos6/fftw-3.3_gcc-4.4.7. > Loading module centos6/gsl-1.16_gcc-4.4.7. > Loading module centos6/hdf5-1.8.11_gcc-4.4.7. > Loading module centos6/netcdf-4.3.0_gcc-4.4.7. > Loading module centos6/R-3.0.2. > Loading required package: minfi > Loading required package: methods > Loading required package: BiocGenerics > Loading required package: parallel > > Attaching package: ââ'¬ËoeBiocGenericsââ'¬â"¢ > > The following objects are masked from ââ'¬Ëoepackage:parallelââ'¬â"¢: > > clusterApply, clusterApplyLB, clusterCall, clusterEvalQ, > clusterExport, clusterMap, parApply, parCapply, parLapply, > parLapplyLB, parRapply, parSapply, parSapplyLB > > The following object is masked from ââ'¬Ëoepackage:statsââ'¬â"¢: > > xtabs > > The following objects are masked from ââ'¬Ëoepackage:baseââ'¬â"¢: > > anyDuplicated, append, as.data.frame, as.vector, cbind, colnames, > duplicated, eval, evalq, Filter, Find, get, intersect, is.unsorted, > lapply, Map, mapply, match, mget, order, paste, pmax, pmax.int, > pmin, pmin.int, Position, rank, rbind, Reduce, rep.int, rownames, > sapply, setdiff, sort, table, tapply, union, unique, unlist > > Loading required package: Biobase > Welcome to Bioconductor > > Vignettes contain introductory material; view with > 'browseVignettes()'. To cite Bioconductor, see > 'citation("Biobase")', and for packages 'citation("pkgname")'. > > Loading required package: lattice > Loading required package: reshape > Loading required package: plyr > > Attaching package: ââ'¬Ëoereshapeââ'¬â"¢ > > The following objects are masked from ââ'¬Ëoepackage:plyrââ'¬â"¢: > > rename, round_any > > Loading required package: GenomicRanges > Loading required package: IRanges > > Attaching package: ââ'¬ËoeIRangesââ'¬â"¢ > > The following objects are masked from ââ'¬Ëoepackage:reshapeââ'¬â"¢: > > expand, rename > > The following objects are masked from ââ'¬Ëoepackage:plyrââ'¬â"¢: > > desc, rename > > Loading required package: XVector > > Attaching package: ââ'¬ËoeXVectorââ'¬â"¢ > > The following object is masked from ââ'¬Ëoepackage:plyrââ'¬â"¢: > > compact > > Loading required package: Biostrings > Loading required package: bumphunter > Loading required package: foreach > Loading required package: iterators > Loading required package: locfit > locfit 1.5-9.1 2013-03-22 > > Attaching package: ââ'¬Ëoelocfitââ'¬â"¢ > > The following objects are masked from ââ'¬Ëoepackage:GenomicRangesââ'¬â"¢: > > left, right > > Loading required package: IlluminaHumanMethylation450kmanifest > Loading required package: IlluminaHumanMethylation450kanno.ilmn12.hg19 > [bumphunterEngine] Parallelizing using 64 workers/cores (backend: > doParallel, version: 1.0.6). > [bumphunterEngine] Computing coefficients. > [bumphunterEngine] Performing 1000 permutations. > [bumphunterEngine] Computing marginal permutation p-values. > [bumphunterEngine] cutoff: 0.05 > [bumphunterEngine] Finding regions. > [bumphunterEngine] Found 175279 bumps. > [bumphunterEngine] Computing regions for each permutation. > Loading required package: rngtools > Loading required package: pkgmaker > Loading required package: registry > > Attaching package: ââ'¬Ëoepkgmakerââ'¬â"¢ > > The following object is masked from ââ'¬Ëoepackage:IRangesââ'¬â"¢: > > new2 > > Error in mcfork() : > unable to fork, possible reason: Cannot allocate memory > Calls: bumphunter ... %dopar% -> <anonymous> -> mclapply -> lapply -> FUN > -> mcfork > Execution halted > > > -- > Sent via the guest posting facility at bioconductor.org. > > _______________________________________________ > Bioconductor mailing list > Bioconductor@r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: > http://news.gmane.org/gmane.science.biology.informatics.conductor > [[alternative HTML version deleted]]
ADD COMMENTlink written 3.6 years ago by Tim Triche4.2k
Hi, Not related to the methylation analysis at all, but: On Sun, Apr 13, 2014 at 4:31 PM, Tim Triche, Jr. <tim.triche at="" gmail.com=""> wrote: > Send your sessionInfo(). > > While you're doing that, here are some suggestions: > > 1) use fewer processes. minfi will kill a machine with 64GB of RAM using > as few as 4 processes for bumphunting. Plus, you probably want to use > doMC() instead of doParallel(). Try setting options("mc.cores") to > something like 4 and gradually increasing it as you successfully run > analyses on this machine/cluster/whatever. I'm curious why you'd suggest that? I thought the general consensus was that the `multicore` package has "passed the baton" to the `parallel` package, so people who want to use "it" via `foreach` should then prefer doParallel ... no? -steve -- Steve Lianoglou Computational Biologist Genentech
ADD REPLYlink written 3.6 years ago by Steve Lianoglou12k
I just checked and they both use the 'parallel' package as a backend. In the past, I've had issues with doParallel locking up on seemingly trivial jobs, whereas I've never had that happen with doMC. This may have long since been resolved, but once bitten, twice shy. Statistics is the grammar of science. Karl Pearson <http: en.wikipedia.org="" wiki="" the_grammar_of_science=""> On Sun, Apr 13, 2014 at 10:23 PM, Steve Lianoglou <lianoglou.steve@gene.com>wrote: > Hi, > > Not related to the methylation analysis at all, but: > > On Sun, Apr 13, 2014 at 4:31 PM, Tim Triche, Jr. <tim.triche@gmail.com> > wrote: > > Send your sessionInfo(). > > > > While you're doing that, here are some suggestions: > > > > 1) use fewer processes. minfi will kill a machine with 64GB of RAM using > > as few as 4 processes for bumphunting. Plus, you probably want to use > > doMC() instead of doParallel(). Try setting options("mc.cores") to > > something like 4 and gradually increasing it as you successfully run > > analyses on this machine/cluster/whatever. > > I'm curious why you'd suggest that? I thought the general consensus > was that the `multicore` package has "passed the baton" to the > `parallel` package, so people who want to use "it" via `foreach` > should then prefer doParallel ... no? > > -steve > > -- > Steve Lianoglou > Computational Biologist > Genentech > [[alternative HTML version deleted]]
ADD REPLYlink written 3.6 years ago by Tim Triche4.2k
Tim, On Mon, Apr 14, 2014 at 11:37 AM, Tim Triche, Jr. <tim.triche at="" gmail.com=""> wrote: > I just checked and they both use the 'parallel' package as a backend. In > the past, I've had issues with doParallel locking up on seemingly trivial > jobs, whereas I've never had that happen with doMC. This may have long > since been resolved, but once bitten, twice shy. Since we're talking about methylation here, I think it's important to remember that the third time is the CHARM ... ... I think my job here is done. -steve -- Steve Lianoglou Computational Biologist Genentech
ADD REPLYlink written 3.6 years ago by Steve Lianoglou12k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.2.0
Traffic: 141 users visited in the last hour