I'm building an aws cluster with bioconductor and I have to install some packages from CRAN prior to the start of the run. I'm using doSNOW for the parallel tasks. When I start the run I receive the error... "X Package not found" where X is my installed packages. If I run on a single multicore system this error is not received. I tried logging into each node and installing locally but I still receive the same error. Any ideas? Thanks -- output of sessionInfo(): "X Package not found" -- Sent via the guest posting facility at bioconductor.org.

On Mon, Jul 15, 2013 at 7:24 AM, John [guest] <guest at="" bioconductor.org=""> wrote: > > I'm building an aws cluster with bioconductor and I have to install some packages from CRAN prior to the start of the run. I'm using doSNOW for the parallel tasks. When I start the run I receive the error... "X Package not found" where X is my installed packages. If I run on a single multicore system this error is not received. > > I tried logging into each node and installing locally but I still receive the same error. > > Any ideas? I think I need more information. What is the command or code you are using to install packages on all nodes in the cluster? How exactly did you start up the cluster? Dan > Thanks > > -- output of sessionInfo(): > > "X Package not found" > > -- > Sent via the guest posting facility at bioconductor.org. > > _______________________________________________ > Bioconductor mailing list > Bioconductor at r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor

Dan Tenenbaum <dtenenba at="" ...=""> writes: > > On Mon, Jul 15, 2013 at 7:24 AM, John [guest] <guest at="" ...=""> wrote: > > > > I'm building an aws cluster with bioconductor and I have to install some packages from CRAN prior to the > start of the run. I'm using doSNOW for the parallel tasks. When I start the run I receive the error... "X > Package not found" where X is my installed packages. If I run on a single multicore system this error is not received. > > > > I tried logging into each node and installing locally but I still receive the same error. > > > > Any ideas? > > I think I need more information. What is the command or code you are > using to install packages on all nodes in the cluster? > > How exactly did you start up the cluster? > > Dan > > > Thanks > > > > -- output of sessionInfo(): > > > > "X Package not found" > > > > -- > > Sent via the guest posting facility at bioconductor.org. > > > > _______________________________________________ > > Bioconductor mailing list > > Bioconductor at ... > > https://stat.ethz.ch/mailman/listinfo/bioconductor > > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor > > _______________________________________________ > Bioconductor mailing list > Bioconductor at ... > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor > > Dan, I'm starting the system using the link "Start parallel Cluster with ssh access" on the Bioconductor - Cloud AMI web page. I create the cluster in R with the "cl <- makePSOCKcluster" as specified on the same page (by reading the host file...). The cluster starts normally and I can see that is contains all of the nodes. I'm exporting the libraries and data: clusterEvalQ(cl, library(the library I need)) clusterExport(cl, c('data', 'i', 'need')) registerDoSNOW(cl) When I execute I get the error "The library X is not found". If I run this on the local cores only... no problem. I did try and ssh into each node and install the library locally then went back to the master to execute and the same thing happened. Thanks for the help... I really appreciate it, John

Dan, I'm just using the standard bioconductor aws ami and I'm installing some extra packages from CRAN. Working from R when I'm running a doSNOW cluster. I see that all of the nodes show up in the cluster, and I'm exporting the libraries as required. Unfortunately, when I run a job I receive an error that the packages are not found (it works if I use only local nodes). I have tried to load the required packages on each node... still no luck.... Thank, John -------- Original Message -------- Subject: Re: [BioC] AWS cluster not finding installed R packages From: Dan Tenenbaum <[1]dtenenba at fhcrc.org> Date: Mon, July 15, 2013 7:37 am To: "John [guest]" <[2]guest at bioconductor.org> Cc: "[3]bioconductor at r-project.org" <[4]bioconductor at r-project.org>, [5]john at enginetix.com On Mon, Jul 15, 2013 at 7:24 AM, John [guest] <[6]guest at bioconductor.org> wrote: > > I'm building an aws cluster with bioconductor and I have to install some packages from CRAN prior to the start of the run. I'm using doSNOW for the parallel tasks. When I start the run I receive the error... "X Package not found" where X is my installed packages. If I run on a single multicore system this error is not received. > > I tried logging into each node and installing locally but I still receive the same error. > > Any ideas? I think I need more information. What is the command or code you are using to install packages on all nodes in the cluster? How exactly did you start up the cluster? Dan > Thanks > > -- output of sessionInfo(): > > "X Package not found" > > -- > Sent via the guest posting facility at [7]bioconductor.org. > > _______________________________________________ > Bioconductor mailing list > [8]Bioconductor at r-project.org > [9]https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: [10]http://news.gmane.org/gmane.science.biology.informatics.conductor References 1. mailto:dtenenba at fhcrc.org 2. mailto:guest at bioconductor.org 3. mailto:bioconductor at r-project.org 4. mailto:bioconductor at r-project.org 5. mailto:john at enginetix.com 6. mailto:guest at bioconductor.org 7. http://bioconductor.org/ 8. mailto:Bioconductor at r-project.org 9. https://stat.ethz.ch/mailman/listinfo/bioconductor 10. http://news.gmane.org/gmane.science.biology.informatics.conductor

On Fri, Jul 19, 2013 at 10:56 AM, <john at="" enginetix.com=""> wrote: > Dan, > > I'm just using the standard bioconductor aws ami and I'm installing some > extra packages from CRAN. Working from R when I'm running a doSNOW cluster. > I see that all of the nodes show up in the cluster, and I'm exporting the > libraries as required. Unfortunately, when I run a job I receive an error > that the packages are not found (it works if I use only local nodes). I have > tried to load the required packages on each node... still no luck.... > Again, difficult to say what is going on without more information, such as a full transcript of your session. Also which AMI are you starting (which BioC version?). Dan > Thank, > John > > -------- Original Message -------- > Subject: Re: [BioC] AWS cluster not finding installed R packages > From: Dan Tenenbaum <dtenenba at="" fhcrc.org=""> > Date: Mon, July 15, 2013 7:37 am > To: "John [guest]" <guest at="" bioconductor.org=""> > Cc: "bioconductor at r-project.org" <bioconductor at="" r-project.org="">, > john at enginetix.com > > On Mon, Jul 15, 2013 at 7:24 AM, John [guest] <guest at="" bioconductor.org=""> > wrote: >> >> I'm building an aws cluster with bioconductor and I have to install some >> packages from CRAN prior to the start of the run. I'm using doSNOW for the >> parallel tasks. When I start the run I receive the error... "X Package not >> found" where X is my installed packages. If I run on a single multicore >> system this error is not received. >> >> I tried logging into each node and installing locally but I still receive >> the same error. >> >> Any ideas? > > I think I need more information. What is the command or code you are > using to install packages on all nodes in the cluster? > > How exactly did you start up the cluster? > > Dan > > >> Thanks >> >> -- output of sessionInfo(): >> >> "X Package not found" >> >> -- >> Sent via the guest posting facility at bioconductor.org. >> >> _______________________________________________ >> Bioconductor mailing list >> Bioconductor at r-project.org >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> Search the archives: >> http://news.gmane.org/gmane.science.biology.informatics.conductor

On Fri, Jul 19, 2013 at 11:45 AM, john <john at="" enginetix.com=""> wrote: > Thanks for the help..... > > I'm using bioc 2.11 (link to start a parallel cluster with ssh on the aws > web page). Is there a process for installing optional R packages from CRAN? > > Sorry for the lack of info.... I'm still coming up to speed.... > Please keep the bioconductor list in the loop so all can benefit from the discussion. Yes, you can use biocLite() or install.packages() to install packages from CRAN. You will also need to install required packages on each cluster node; the method for doing this varies with the different ways there are to start a cluster, but with doSNOW, assuming you've already started a cluster called 'cl', and the packages you want to install are survival and abc, you'd do something like: clusterEvalQ(cl, {library(BiocInstaller); biocLite(c("survival", "abc"))}) Then you can run your parallel computation. Dan > > Sent from my Verizon Wireless Phone > > > ----- Reply message ----- > From: "Dan Tenenbaum" <dtenenba at="" fhcrc.org=""> > To: <john at="" enginetix.com=""> > Cc: "John [guest]" <guest at="" bioconductor.org="">, "bioconductor at r-project.org" > <bioconductor at="" r-project.org=""> > Subject: [BioC] AWS cluster not finding installed R packages > Date: Fri, Jul 19, 2013 1:58 pm > > > On Fri, Jul 19, 2013 at 10:56 AM, <john at="" enginetix.com=""> wrote: >> Dan, >> >> I'm just using the standard bioconductor aws ami and I'm installing some >> extra packages from CRAN. Working from R when I'm running a doSNOW >> cluster. >> I see that all of the nodes show up in the cluster, and I'm exporting the >> libraries as required. Unfortunately, when I run a job I receive an error >> that the packages are not found (it works if I use only local nodes). I >> have >> tried to load the required packages on each node... still no luck.... >> > > Again, difficult to say what is going on without more information, > such as a full transcript of your session. Also which AMI are you > starting (which BioC version?). > > Dan > > >> Thank, >> John >> >> -------- Original Message -------- >> Subject: Re: [BioC] AWS cluster not finding installed R packages >> From: Dan Tenenbaum <dtenenba at="" fhcrc.org=""> >> Date: Mon, July 15, 2013 7:37 am >> To: "John [guest]" <guest at="" bioconductor.org=""> >> Cc: "bioconductor at r-project.org" <bioconductor at="" r-project.org="">, >> john at enginetix.com >> >> On Mon, Jul 15, 2013 at 7:24 AM, John [guest] <guest at="" bioconductor.org=""> >> wrote: >>> >>> I'm building an aws cluster with bioconductor and I have to install some >>> packages from CRAN prior to the start of the run. I'm using doSNOW for >>> the >>> parallel tasks. When I start the run I receive the error... "X Package >>> not >>> found" where X is my installed packages. If I run on a single multicore >>> system this error is not received. >>> >>> I tried logging into each node and installing locally but I still receive >>> the same error. >>> >>> Any ideas? >> >> I think I need more information. What is the command or code you are >> using to install packages on all nodes in the cluster? >> >> How exactly did you start up the cluster? >> >> Dan >> >> >>> Thanks >>> >>> -- output of sessionInfo(): >>> >>> "X Package not found" >>> >>> -- >>> Sent via the guest posting facility at bioconductor.org. >>> >>> _______________________________________________ >>> Bioconductor mailing list >>> Bioconductor at r-project.org >>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>> Search the archives: >>> http://news.gmane.org/gmane.science.biology.informatics.conductor >