Offline Installation of Packages and Dependencies
1
0
Entering edit mode
@nathan-bihlmeyer-5641
Last seen 9.7 years ago
Hello all, I am currently working on a HPC cluster that for security reasons does not have access to the internet on the compute nodes. Therefore, I wish to quickly download package source archives using the head node, then compile them on a compute node. Other details: The head node does not have same shared libraries as the compute nodes, so I couldn't compile on that node even if I wanted to. All the nodes use the same network-based file system for most files (/home etc...). I am using my own compiled version of R 2.15.2. For non-Bioconductor packages, I have been using the following scripts: ### On the head node #!/usr/bin/env Rscript args = commandArgs(T) rh = R.home() tmp = paste(rh,"/tmp",sep="") if(!file.exists(tmp)) { dir.create(tmp) } pkg = utils:::getDependencies(args,,available.packages()) download.packages(pkg, tmp) ### On the compute node #!/usr/bin/env Rscript rh = R.home() tmp = paste(rh,"/tmp",sep="") if(!file.exists(tmp)) { quit() } pkg = list.files(tmp,full.names=T) install.packages(pkg,repo=NULL) file.remove(pkg) They work well. What I need now is a way to do the same thing with Bioconductor packages. I have looked into pkgDepTools; however, the package requires a complete dependency graph to be built every time the script is run. This takes too long and uses too many resources and therefore is not an option on the head node. Also, I can not find a way to produce download urls using that package. Thanks for your help, Nathan Bihlmeyer Human Genetics School of Medicine Johns Hopkins University
graph graph • 4.0k views
ADD COMMENT
0
Entering edit mode
@kasper-daniel-hansen-2979
Last seen 11 months ago
United States
Forgot to reply all Most of the R commands for managing packages has a repos argument. You can do library(BiocInstaller) install.packages(repos = biocinstallRepos()) Be careful about R versions here, if nodes are different from head. Also, some commands may not accept a repos argument, but instead a contriburl argument which you can construct using contrib.url(biocinstallRepos()) If you start to grab some of the annotation / experiment data packages (especially the genomes), watch out that they are much much bigger than a typical CRAN package. Better make sure /tmp is big enough. Kasper On Mon, Dec 3, 2012 at 2:10 PM, Nathan Bihlmeyer <nbihlme1 at="" jhmi.edu=""> wrote: > Hello all, > > I am currently working on a HPC cluster that for security reasons does > not have access to the internet on the compute nodes. Therefore, I > wish to quickly download package source archives using the head node, > then compile them on a compute node. > > Other details: The head node does not have same shared libraries as > the compute nodes, so I couldn't compile on that node even if I wanted > to. All the nodes use the same network-based file system for most > files (/home etc...). I am using my own compiled version of R 2.15.2. > > For non-Bioconductor packages, I have been using the following scripts: > > ### On the head node > #!/usr/bin/env Rscript > args = commandArgs(T) > rh = R.home() > tmp = paste(rh,"/tmp",sep="") > if(!file.exists(tmp)) { > dir.create(tmp) > } > pkg = utils:::getDependencies(args,,available.packages()) > download.packages(pkg, tmp) > > ### On the compute node > #!/usr/bin/env Rscript > rh = R.home() > tmp = paste(rh,"/tmp",sep="") > if(!file.exists(tmp)) { > quit() > } > pkg = list.files(tmp,full.names=T) > install.packages(pkg,repo=NULL) > file.remove(pkg) > > They work well. What I need now is a way to do the same thing with > Bioconductor packages. I have looked into pkgDepTools; however, the > package requires a complete dependency graph to be built every time > the script is run. This takes too long and uses too many resources and > therefore is not an option on the head node. Also, I can not find a > way to produce download urls using that package. > > Thanks for your help, > Nathan Bihlmeyer > Human Genetics > School of Medicine > Johns Hopkins University > > _______________________________________________ > Bioconductor mailing list > Bioconductor at r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor
ADD COMMENT
0
Entering edit mode
Thank you Kasper. I have got everything working the way I want now. In case anyone wants to know, here are my scripts: ### On the head node #!/usr/bin/env Rscript args = commandArgs(T) rh = R.home() tmp = paste(rh,"/tmp",sep="") if(!file.exists(tmp)) { dir.create(tmp) } source("http://bioconductor.org/biocLite.R") repos = c(biocinstallRepos(),getOption("repos")) pkg = utils:::getDependencies(args,,available.packages(contriburl=cont rib.url(repos))) order = download.packages(pkg, tmp, repos=repos)[,2] write.table(order,paste(tmp,"/pkg_install_order.list",sep=""),col.name s=F,row.names=F,quote=F,sep="\t") ### On the compute node #!/usr/bin/env Rscript rh = R.home() tmp = paste(rh,"/tmp",sep="") if(!file.exists(tmp)) { quit() } pkg = read.table(paste(tmp,"/pkg_install_order.list",sep=""),sep="\t") pkg = as.vector(t(pkg)) install.packages(pkg,repo=NULL) file.remove(pkg) file.remove(paste(tmp,"/pkg_install_order.list",sep="")) Thanks, Nathan Bihlmeyer Human Genetics School of Medicine Johns Hopkins University On Tue, Dec 4, 2012 at 11:19 AM, Kasper Daniel Hansen <kasperdanielhansen at="" gmail.com=""> wrote: > Forgot to reply all > > Most of the R commands for managing packages has a repos argument. You can do > library(BiocInstaller) > install.packages(repos = biocinstallRepos()) > Be careful about R versions here, if nodes are different from head. > Also, some commands may not accept a repos argument, but instead a > contriburl argument which you can construct using > contrib.url(biocinstallRepos()) > If you start to grab some of the annotation / experiment data packages > (especially the genomes), watch out that they are much much bigger > than a typical CRAN package. Better make sure /tmp is big enough. > > Kasper > > On Mon, Dec 3, 2012 at 2:10 PM, Nathan Bihlmeyer <nbihlme1 at="" jhmi.edu=""> wrote: >> Hello all, >> >> I am currently working on a HPC cluster that for security reasons does >> not have access to the internet on the compute nodes. Therefore, I >> wish to quickly download package source archives using the head node, >> then compile them on a compute node. >> >> Other details: The head node does not have same shared libraries as >> the compute nodes, so I couldn't compile on that node even if I wanted >> to. All the nodes use the same network-based file system for most >> files (/home etc...). I am using my own compiled version of R 2.15.2. >> >> For non-Bioconductor packages, I have been using the following scripts: >> >> ### On the head node >> #!/usr/bin/env Rscript >> args = commandArgs(T) >> rh = R.home() >> tmp = paste(rh,"/tmp",sep="") >> if(!file.exists(tmp)) { >> dir.create(tmp) >> } >> pkg = utils:::getDependencies(args,,available.packages()) >> download.packages(pkg, tmp) >> >> ### On the compute node >> #!/usr/bin/env Rscript >> rh = R.home() >> tmp = paste(rh,"/tmp",sep="") >> if(!file.exists(tmp)) { >> quit() >> } >> pkg = list.files(tmp,full.names=T) >> install.packages(pkg,repo=NULL) >> file.remove(pkg) >> >> They work well. What I need now is a way to do the same thing with >> Bioconductor packages. I have looked into pkgDepTools; however, the >> package requires a complete dependency graph to be built every time >> the script is run. This takes too long and uses too many resources and >> therefore is not an option on the head node. Also, I can not find a >> way to produce download urls using that package. >> >> Thanks for your help, >> Nathan Bihlmeyer >> Human Genetics >> School of Medicine >> Johns Hopkins University >> >> _______________________________________________ >> Bioconductor mailing list >> Bioconductor at r-project.org >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor
ADD REPLY

Login before adding your answer.

Traffic: 371 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6