Question: Generating random gene lists: does sample/resample generate random sets
0
gravatar for Scott Ochsner
11.2 years ago by
Scott Ochsner300
Scott Ochsner300 wrote:
Thomas, I wanted to asses the performance of random gene lists which do not have any overlap with myCuratedList hence the step to remove them from the universe of possible genes prior to random gene selection. If I leave the curated genes in, random lists could potentially be produced with significant similarity to myCuratedList. I'm interested in the chance occurrence of unique gene lists with similar classification performance as myCuratedList. I certainly have an open mind with this point if others can come up good reasons why this may be a bad idea. Scott ________________________________ From: Thomas Hampton [mailto:Thomas.H.Hampton@Dartmouth.EDU] Sent: Wed 9/10/2008 3:40 PM To: Ochsner, Scott A Cc: bioconductor@stat.math.ethz.ch Subject: Re: [BioC] Generating random gene lists: does sample/resample generate random sets I would not have taken the curated list out. That strikes me as a significant bias. Am I missing something? Tom On Sep 10, 2008, at 4:03 PM, Ochsner, Scott A wrote: > Dear BioC, > > I would like feedback as to the appropriateness of the following > procedure to produce a set of 1000 random gene lists, each list of > length 2000. The idea is to use the set of random gene lists to > assess how often random gene lists of size x can reproduce or > improve the classification performance of > myCuratedList. > > > #remove myCuratedList from the universe of possible genes. The > "eset" object is your standard ExpressionSet object. >> length(myCuratedList) > [1] 2000 >> Index<-setdiff(1:length(rownames(exprs(eset))),myCuratedList) >> length(Index) > [1] 20277 > #generate 1000 random gene lists using the genes in Index. The > code for resample is taken from the help pages for sample. > >> randomMatrix<-replicate(1000,resample(index,2000)) >> dim(randomMatrix) > [1] 2000 1000 > > > I've verified that each column does not contain repeated genes as > should be the case with resample without replacement. > > Is there a standard procedure for doing the above or is what I've > done kosher? > > > Scott A. Ochsner, Ph.D. > NURSA Bioinformatics > Molecular and Cellular Biology > Baylor College of Medicine > Houston, TX. 77030 > phone: 713-798-6227 > > _______________________________________________ > Bioconductor mailing list > Bioconductor@stat.math.ethz.ch > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/ > gmane.science.biology.informatics.conductor [[alternative HTML version deleted]]
classification • 401 views
ADD COMMENTlink modified 11.2 years ago • written 11.2 years ago by Scott Ochsner300
Answer: Generating random gene lists: does sample/resample generate random sets
0
gravatar for Sean Davis
11.2 years ago by
Sean Davis21k
United States
Sean Davis21k wrote:
On Thu, Sep 11, 2008 at 12:28 AM, Ochsner, Scott A <sochsner at="" bcm.tmc.edu=""> wrote: > Thomas, > > I wanted to asses the performance of random gene lists which do not have any overlap with myCuratedList hence the step to remove them from the universe of possible genes prior to random gene selection. If I leave the curated genes in, random lists could potentially be produced with significant > similarity to myCuratedList. I'm interested in the chance occurrence of unique gene lists with similar classification performance as myCuratedList. I certainly have an open mind with this point if others can come up good reasons why this may be a bad idea. > I agree with Tom, here. You must include all the genes that were originally included when you produced "myCuratedList". Not doing so makes precludes drawing any conclusions from the randomization results. Sean > From: Thomas Hampton [mailto:Thomas.H.Hampton at Dartmouth.EDU] > Sent: Wed 9/10/2008 3:40 PM > To: Ochsner, Scott A > Cc: bioconductor at stat.math.ethz.ch > Subject: Re: [BioC] Generating random gene lists: does sample/resample generate random sets > > > > I would not have taken the curated list out. That strikes me as > a significant bias. Am I missing something? > > Tom > > On Sep 10, 2008, at 4:03 PM, Ochsner, Scott A wrote: > >> Dear BioC, >> >> I would like feedback as to the appropriateness of the following >> procedure to produce a set of 1000 random gene lists, each list of >> length 2000. The idea is to use the set of random gene lists to >> assess how often random gene lists of size x can reproduce or >> improve the classification performance of >> myCuratedList. >> >> >> #remove myCuratedList from the universe of possible genes. The >> "eset" object is your standard ExpressionSet object. >>> length(myCuratedList) >> [1] 2000 >>> Index<-setdiff(1:length(rownames(exprs(eset))),myCuratedList) >>> length(Index) >> [1] 20277 >> #generate 1000 random gene lists using the genes in Index. The >> code for resample is taken from the help pages for sample. >> >>> randomMatrix<-replicate(1000,resample(index,2000)) >>> dim(randomMatrix) >> [1] 2000 1000 >> >> >> I've verified that each column does not contain repeated genes as >> should be the case with resample without replacement. >> >> Is there a standard procedure for doing the above or is what I've >> done kosher? >> >> >> Scott A. Ochsner, Ph.D. >> NURSA Bioinformatics >> Molecular and Cellular Biology >> Baylor College of Medicine >> Houston, TX. 77030 >> phone: 713-798-6227 >> >> _______________________________________________ >> Bioconductor mailing list >> Bioconductor at stat.math.ethz.ch >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> Search the archives: http://news.gmane.org/ >> gmane.science.biology.informatics.conductor > > > > > [[alternative HTML version deleted]] > > _______________________________________________ > Bioconductor mailing list > Bioconductor at stat.math.ethz.ch > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor >
ADD COMMENTlink written 11.2 years ago by Sean Davis21k
Answer: Generating random gene lists: does sample/resample generate random sets
0
gravatar for Scott Ochsner
11.2 years ago by
Scott Ochsner300
Scott Ochsner300 wrote:
Thomas, Sean, I see your reasoning and will leave the curated list of genes in the universe prior to random list generation. However, what about the following scenario? Lets say the genes are left in, 1000 random gene lists are generated, and each of these are tested for their classification performance. If I repeat this whole process say 1000 times and find that on average 70 random gene lists out of 1000 perform equally well as myCuratedList in sample classification. Not a ringing endorsement for myCuratedList. Yet, there is the possibility that the random gene lists which perform equally well have significant overlap with myCuratedList and are essentially the same as myCuratedList. How do I adjust the probability to account for this? Scott -----Original Message----- From: Thomas Hampton [mailto:Thomas.H.Hampton@Dartmouth.EDU] Sent: Thursday, September 11, 2008 7:36 AM To: Ochsner, Scott A Subject: Re: [BioC] Generating random gene lists: does sample/resample generate random sets I kind of get your reasoning, but not quite. Does this analogy make sense? Suppose I shuffle a deck and draw a full house off the top, specifically, a pair of 2s and three 10s. If I want to know how often a full house would happen by chance, I would not set these five cards aside to do the simulation, because the resulting deck of 47 cards is a bit less likely to come up with a full house than a real deck of 52. In a more extreme version of your test, suppose your curated list constituted a larger percentage of the entire gene list. Suppose you removed 75% and ran your simulations on the remaining 25%. We could not offer that 25% as representative of the whole, could we? Cheers T On Sep 11, 2008, at 12:28 AM, Ochsner, Scott A wrote: > Thomas, > > I wanted to asses the performance of random gene lists which do not > have any overlap with myCuratedList hence the step to remove them from > the universe of possible genes prior to random gene selection. If I > leave the curated genes in, random lists could potentially be produced > with significant similarity to myCuratedList. I'm interested in the > chance occurrence of unique gene lists with similar classification > performance as myCuratedList. I certainly have an open mind with this > point if others can come up good reasons why this may be a bad idea. > > Scott > > ________________________________ > > From: Thomas Hampton [mailto:Thomas.H.Hampton at Dartmouth.EDU] > Sent: Wed 9/10/2008 3:40 PM > To: Ochsner, Scott A > Cc: bioconductor at stat.math.ethz.ch > Subject: Re: [BioC] Generating random gene lists: does sample/ > resample generate random sets > > > > I would not have taken the curated list out. That strikes me as a > significant bias. Am I missing something? > > Tom > > On Sep 10, 2008, at 4:03 PM, Ochsner, Scott A wrote: > >> Dear BioC, >> >> I would like feedback as to the appropriateness of the following >> procedure to produce a set of 1000 random gene lists, each list of >> length 2000. The idea is to use the set of random gene lists to >> assess how often random gene lists of size x can reproduce or improve >> the classification performance of myCuratedList. >> >> >> #remove myCuratedList from the universe of possible genes. The >> "eset" object is your standard ExpressionSet object. >>> length(myCuratedList) >> [1] 2000 >>> Index<-setdiff(1:length(rownames(exprs(eset))),myCuratedList) >>> length(Index) >> [1] 20277 >> #generate 1000 random gene lists using the genes in Index. The code >> for resample is taken from the help pages for sample. >> >>> randomMatrix<-replicate(1000,resample(index,2000)) >>> dim(randomMatrix) >> [1] 2000 1000 >> >> >> I've verified that each column does not contain repeated genes as >> should be the case with resample without replacement. >> >> Is there a standard procedure for doing the above or is what I've >> done kosher? >> >> >> Scott A. Ochsner, Ph.D. >> NURSA Bioinformatics >> Molecular and Cellular Biology >> Baylor College of Medicine >> Houston, TX. 77030 >> phone: 713-798-6227 >> >> _______________________________________________ >> Bioconductor mailing list >> Bioconductor at stat.math.ethz.ch >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> Search the archives: http://news.gmane.org/ >> gmane.science.biology.informatics.conductor > > > > > [[alternative HTML version deleted]] > > _______________________________________________ > Bioconductor mailing list > Bioconductor at stat.math.ethz.ch > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/ > gmane.science.biology.informatics.conductor
ADD COMMENTlink written 11.2 years ago by Scott Ochsner300
On Thu, Sep 11, 2008 at 10:59 AM, Ochsner, Scott A <sochsner at="" bcm.tmc.edu=""> wrote: > Thomas, Sean, > > I see your reasoning and will leave the curated list of genes in the universe prior to random list generation. However, what about the following scenario? Lets say the genes are left in, 1000 random gene lists are generated, and each of these are tested for their classification performance. If I > repeat this whole process say 1000 times and find that on average 70 random gene lists out of 1000 perform equally well as myCuratedList in sample classification. Not a ringing endorsement for myCuratedList. Yet, there is the possibility that the random gene lists which perform equally well have > significant overlap with myCuratedList and are essentially the same as myCuratedList. How do I adjust the probability to account for this? > Hi, Scott. That is exactly the test that you want to do and, yes, it could have a just the outcome that you suggest, in which case you do not have terribly strong evidence for your gene list being better than random. But stacking the deck (or depleting the deck, as the case may be) is a definite no-no. The results obtained by leaving your gene list out are not interpretable, so don't bother comparing the two different methods, one leaving your gene list in and one taking it out; the latter is simply not valid. Sean > Scott > -----Original Message----- > From: Thomas Hampton [mailto:Thomas.H.Hampton at Dartmouth.EDU] > Sent: Thursday, September 11, 2008 7:36 AM > To: Ochsner, Scott A > Subject: Re: [BioC] Generating random gene lists: does sample/resample generate random sets > > I kind of get your reasoning, but not quite. > > Does this analogy make sense? > > Suppose I shuffle a deck and draw a full house off the top, specifically, a pair of 2s and three 10s. > > If I want to know how often a full house would happen by chance, I would not set these five cards aside to do the simulation, because the resulting deck of 47 cards is a bit less likely to come up with a full house than a real deck of 52. > > In a more extreme version of your test, suppose your curated list constituted a larger percentage of the entire gene list. Suppose you removed 75% and ran your simulations on the remaining 25%. We could not offer that 25% as representative of the whole, could we? > > Cheers > > T > On Sep 11, 2008, at 12:28 AM, Ochsner, Scott A wrote: > >> Thomas, >> >> I wanted to asses the performance of random gene lists which do not >> have any overlap with myCuratedList hence the step to remove them from >> the universe of possible genes prior to random gene selection. If I >> leave the curated genes in, random lists could potentially be produced >> with significant similarity to myCuratedList. I'm interested in the >> chance occurrence of unique gene lists with similar classification >> performance as myCuratedList. I certainly have an open mind with this >> point if others can come up good reasons why this may be a bad idea. >> >> Scott >> >> ________________________________ >> >> From: Thomas Hampton [mailto:Thomas.H.Hampton at Dartmouth.EDU] >> Sent: Wed 9/10/2008 3:40 PM >> To: Ochsner, Scott A >> Cc: bioconductor at stat.math.ethz.ch >> Subject: Re: [BioC] Generating random gene lists: does sample/ >> resample generate random sets >> >> >> >> I would not have taken the curated list out. That strikes me as a >> significant bias. Am I missing something? >> >> Tom >> >> On Sep 10, 2008, at 4:03 PM, Ochsner, Scott A wrote: >> >>> Dear BioC, >>> >>> I would like feedback as to the appropriateness of the following >>> procedure to produce a set of 1000 random gene lists, each list of >>> length 2000. The idea is to use the set of random gene lists to >>> assess how often random gene lists of size x can reproduce or improve >>> the classification performance of myCuratedList. >>> >>> >>> #remove myCuratedList from the universe of possible genes. The >>> "eset" object is your standard ExpressionSet object. >>>> length(myCuratedList) >>> [1] 2000 >>>> Index<-setdiff(1:length(rownames(exprs(eset))),myCuratedList) >>>> length(Index) >>> [1] 20277 >>> #generate 1000 random gene lists using the genes in Index. The code >>> for resample is taken from the help pages for sample. >>> >>>> randomMatrix<-replicate(1000,resample(index,2000)) >>>> dim(randomMatrix) >>> [1] 2000 1000 >>> >>> >>> I've verified that each column does not contain repeated genes as >>> should be the case with resample without replacement. >>> >>> Is there a standard procedure for doing the above or is what I've >>> done kosher? >>> >>> >>> Scott A. Ochsner, Ph.D. >>> NURSA Bioinformatics >>> Molecular and Cellular Biology >>> Baylor College of Medicine >>> Houston, TX. 77030 >>> phone: 713-798-6227 >>> >>> _______________________________________________ >>> Bioconductor mailing list >>> Bioconductor at stat.math.ethz.ch >>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>> Search the archives: http://news.gmane.org/ >>> gmane.science.biology.informatics.conductor >> >> >> >> >> [[alternative HTML version deleted]] >> >> _______________________________________________ >> Bioconductor mailing list >> Bioconductor at stat.math.ethz.ch >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> Search the archives: http://news.gmane.org/ >> gmane.science.biology.informatics.conductor > > _______________________________________________ > Bioconductor mailing list > Bioconductor at stat.math.ethz.ch > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor >
ADD REPLYlink written 11.2 years ago by Sean Davis21k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 290 users visited in the last hour