Crispin Miller wrote: > I'm not sure if this is your experience too?: we found we need a big > machine mainly for the normalization/expression summary side of things - > I don't think the package itself should need too much clout to manage > the annotation; it's the arrays themselves that eat up the RAM. Exactly. Well, the cdf package is pretty big as well, although I don't know offhand how much RAM it takes. But you are exactly right - the arrays are the big RAM users here. There are two possible fixes in the pipeline. For the cdf there will soon be the ability to create a hybrid cdf/annotation package based on a SQLite database that will use much less RAM. Because SQLite doesn't have the capabilities that MySQL does (stored procedures, for instance), I don't think this will be able to supercede exonmap. Rather just complement it. On the array side, there is the BufferedMatrix package which puts most of the data on disk (and buffers a small, user adjustable amount of data in RAM). The oligo package already uses BufferedMatrix for the raw data and I imagine could also use BufferedMatrix for ExpressionSets as well. So hopefully we will soon get the analysis of these data back onto more modest computers. Best, Jim > > Cheers, > > Crispin > > -------------------------------------------------------- > > > This email is confidential and intended solely for the use...{{dropped}}

Crispin Miller wrote: > Hi Daniel, Jim, > >> Have you looked at the exonmap package? >> >> IIRC Affy intend iterPLIER to correct for the fact that lots of the > probesets on the exon array don't actually interrogate exons (according > to current >> knowledge of the genome). In fact Wing Wong has recently published a > paper showing that the correlation between the 'core' >> probesets and the 'full' and 'extended' are really bad. > >> IMO, it is better to remove the probesets that we currently think are > either interrogating multiple transcripts or missing the exons > altogether _before_ > computing any expression values, so you don't have > to hope that the statistics are robust enough to ignore spurious signal. > >> One downside to the exonmap package is the fact that you need a 64 bit > linux box with lots of RAM (which presumably you have already, else how > are you >> doing anything with these things?). In addition, you need to install > MySQL and set up the Ensembl core database, as well as the tables for > exonmap. >> However, Michal and Crispin have given pretty detailed instructions > for how to go about doing that (I was able to get set up, and my > knowledge of DBs >> wouldn't fill a thimble). > > I'm not sure if this is your experience too?: we found we need a big > machine mainly for the normalization/expression summary side of things - > I don't think the package itself should need too much clout to manage > the annotation; it's the arrays themselves that eat up the RAM. > > Cheers, > > Crispin I did look into exonmap, but was scared off by the memory (mainly) and setup requirements. Is it possible to use APT (affymetrix power tools) to produce the exon-level summaries and then use the exonmap package for the annotation and analysis? The advantages to using APT is that it uses memory efficient techniques in C++ unlike R. Many thanks Dan -- ************************************************************** Daniel Brewer, Ph.D. Institute of Cancer Research Email: daniel.brewer at icr.ac.uk ************************************************************** The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company Limited by Guarantee, Registered in England under Company No. 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP. This e-mail message is confidential and for use by the addre...{{dropped}}