Multiple test question in micrarray- FDR
1
0
Entering edit mode
Naomi Altman ★ 6.0k
@naomi-altman-380
Last seen 2.9 years ago
United States
Remember that FDR is a rate - i.e. the expected false discovery rate. If the set of genes is changeds, FDR will change because the comparison set is different. This is NOT the same as a p-value, which depends only on the value of the current test statistic. The same thing happens with FWER, because these methods control the probability of making at least one mistake, which clearly depends on which set of tests are performed. --Naomi At 03:11 PM 12/13/2008, Sean Davis wrote: >On Sat, Dec 13, 2008 at 12:36 PM, Wayne Xu <wxu at="" msi.umn.edu=""> wrote: > > Hello, > > I am not sure this is a right place to ask this question, but it is about > > micrarray data analysis: > > > > In two group t test, the multiple test Q values are depending on the total > > number of genes in the test. If I filter the gene list first, for > example, I > > only use those genes that have1.2 fold changes for T test and > multiple test, > > this gene list is much smaller than the total gene list, then the multiple > > test q values are much smaller. > > > > Do you think above is a correct way? People who do not do that way may > > consider the statistical power may be lost? But how much power lost and how > > to calculate the power in this case? > >No, you cannot filter based on fold change. However, you can filter >based on variance or some other measure that does not depend on the >two groups being compared. Anything that filters genes based on >"knowing" the two groups will lead to a biased test. Remember that >filtering removes genes from consideration from further analysis. > >For further details, there are MANY discussions of this topic in the >mailing list. > > > When people report multiple test Q values, they usually do not mention how > > many genes are used in this multiple test. You can get different Q values > > (even use the same method, e.g. Benjamin and Holm adjust method) > in the same > > dataset. Then how can it make sense if the same genes have different Q > > values? > >A good manuscript should describe in detail the preprocessing and >filtering steps, the statistical tests used, and the methods for >correcting for multiple testing. You are correct that many papers do >not do so. > >As for different q-values in the same dataset using different methods, >it is important to note that one should not do an analysis, get a >result, and then, based on that result, go back and redo the analysis >with different parameters to get a "better" result. It is very >important that each step of an analysis (preprocessing, filtering, >testing, multiple-testing correction) be justifiable independent of >the other steps in order for the results to be interpretable. > >Sean > >_______________________________________________ >Bioconductor mailing list >Bioconductor at stat.math.ethz.ch >https://stat.ethz.ch/mailman/listinfo/bioconductor >Search the archives: >http://news.gmane.org/gmane.science.biology.informatics.conductor Naomi S. Altman 814-865-3791 (voice) Associate Professor Dept. of Statistics 814-863-7114 (fax) Penn State University 814-865-1348 (Statistics) University Park, PA 16802-2111
GO Preprocessing GO Preprocessing • 1.1k views
ADD COMMENT
0
Entering edit mode
@wxumsiumnedu-1819
Last seen 9.6 years ago
Dear Naomi, I may have a silly question. I read a few papers on microarray multiple test, I understood what points they were trying to make. But I still have doubts about it. Since now many journal reviewers require the FDR for microarray differential expresses genes in manuscripts, I really want to clear my doubts. 1). The mathematics model is different from the biology model: The typical math model to bring up the multiple test issue is following example: 20 balls in a box with 1 in red and 19 in blue. The possibility of picking up the red ball from the box each time is 1/20, i.e 0.05. If draw 20 times, the chance is 0.05 multiplied by 20 is 1. Suppose the red represents false positive, if draw one time the FDR is 0.05, if 20 times then FDR is 1. People bring this multiple test issue into microarray data analysis. But in microarray, at least two aspects are different from this math model: a). The raw P values are determined by the expression values of samples, not affected by the total number of genes. So it is different from above example of 1 out of 20 is 0.05. b). Pick up a ball and then put it back to the box, you have chance to pick up the exactly same ball twice or more. But in microarray, each genes are tested individually at the same time, and each gene only tested exactly once. They are obviously different. If this math model is the only reason that brought up the multiple test issue in microarray, it may be a misleading (I may be silly, since no one else doubts about multiple test in microarray?) 2). Not make biological sense: Suppose a gene called XYZ has a raw P value of 0.00001 in two group T test, and it was validated by biological test, e.g. RT-PCR. If the micoarray chip has 40,000 genes, then by whatever adjustment FDR method, the adj P-value may be 0.4 or lower or higher. If I use FDR cutoff 0.1, this XYZ gene has higher FDR and is not in my interest positive gene list. OK, now I play a math game, filter gene by variance or other, shrink the gene list to 5000 (since XYZ gene has low P value, suppose it is within the 5000). Then the XYZ has low FDR and in my interest differential gene list. But this is just a math game! The biological reality is XYZ is positive, this positive is determined by, for example 4 control samples and 4 treatment samples, the mean may be big different, and within group variance is very small. and RT-PCR validated. This reality can not be changed by whatever number of genes to be tested. The raw P value is close the biological reality, and it is good to represent the biological reality. The multiple test here just make you feel happier but not a biological sense. FDR is a very useful term in many biological cases. But it seems not a good example here for microarray? Please help to clear it up. Thank you, Wayne -- Naomi Altman wrote: > Remember that FDR is a rate - i.e. the expected false discovery rate. > If the set of genes is changeds, FDR will change because the > comparison set is different. This is NOT the same as a p-value, which > depends only on the value of the current test statistic. > > The same thing happens with FWER, because these methods control the > probability of making at least one mistake, which clearly depends on > which set of tests are performed. > > --Naomi > > At 03:11 PM 12/13/2008, Sean Davis wrote: >> On Sat, Dec 13, 2008 at 12:36 PM, Wayne Xu <wxu at="" msi.umn.edu=""> wrote: >> > Hello, >> > I am not sure this is a right place to ask this question, but it is >> about >> > micrarray data analysis: >> > >> > In two group t test, the multiple test Q values are depending on >> the total >> > number of genes in the test. If I filter the gene list first, for >> example, I >> > only use those genes that have1.2 fold changes for T test and >> multiple test, >> > this gene list is much smaller than the total gene list, then the >> multiple >> > test q values are much smaller. >> > >> > Do you think above is a correct way? People who do not do that way may >> > consider the statistical power may be lost? But how much power lost >> and how >> > to calculate the power in this case? >> >> No, you cannot filter based on fold change. However, you can filter >> based on variance or some other measure that does not depend on the >> two groups being compared. Anything that filters genes based on >> "knowing" the two groups will lead to a biased test. Remember that >> filtering removes genes from consideration from further analysis. >> >> For further details, there are MANY discussions of this topic in the >> mailing list. >> >> > When people report multiple test Q values, they usually do not >> mention how >> > many genes are used in this multiple test. You can get different Q >> values >> > (even use the same method, e.g. Benjamin and Holm adjust method) in >> the same >> > dataset. Then how can it make sense if the same genes have different Q >> > values? >> >> A good manuscript should describe in detail the preprocessing and >> filtering steps, the statistical tests used, and the methods for >> correcting for multiple testing. You are correct that many papers do >> not do so. >> >> As for different q-values in the same dataset using different methods, >> it is important to note that one should not do an analysis, get a >> result, and then, based on that result, go back and redo the analysis >> with different parameters to get a "better" result. It is very >> important that each step of an analysis (preprocessing, filtering, >> testing, multiple-testing correction) be justifiable independent of >> the other steps in order for the results to be interpretable. >> >> Sean >> >> _______________________________________________ >> Bioconductor mailing list >> Bioconductor at stat.math.ethz.ch >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> Search the archives: >> http://news.gmane.org/gmane.science.biology.informatics.conductor > > Naomi S. Altman 814-865-3791 (voice) > Associate Professor > Dept. of Statistics 814-863-7114 (fax) > Penn State University 814-865-1348 (Statistics) > University Park, PA 16802-2111
ADD COMMENT
0
Entering edit mode
The ball model does not apply to microarray studies. (And the probability of drawing the red ball in 20 draws is not 1). But FDR does apply to microarray studies, and so does a less discussed concept, the false nondiscovery rate or FNR. Suppose I take 20 independent samples of mouse liver tissue - same strain, gender ... and hybridize independently to 20 microarrays - any platform. Then arbitrarily divide into 2 groups of size 10. If there are 10,000 genes on the array, you should see 1 gene with p-value .0001or less, 10 genes with p-value .001 or less, 100 genes with p-value .01 or less etc. Now suppose you take the 100 genes with the highest degree of differential expression and do a PCR study with independent samples. You should still have 1 gene which is significant with p=.01 and 5 genes which are significant at p=.05. The problem is - there is no systematic difference between the samples. You have detected noise - i.e. chance variation. If you use the same samples to do your PCR, you may get closer to 100% "significance" for the selected genes, because the variation that caused the false detection will still be in the sample unless it was due only to the hybridization. FDR is an estimate of the excess of significant findings, compared to what is expected by chance. You can reduce FDR greatly by doing independent follow-up studies (on another microarray or on another platform such as PCR). You cannot reduce FDR much by reusing the same samples on a different platform, although you will reduce affects due to technical variation. However, FDR reduces your power to detect differential expression. This means that you will have higher FNR if you use multiple comparisons adjustments. Again, if you do independent follow-up studies, you can reduce FNR. The purpose of the FDR computation is to reduce effort wasted on large gene lists which are mostly reporting noise. But if your genelist is smaller than you think is reasonable, you may certainly follow up a larger set of genes and sorting by p-value will give you the most reasonable set of genes to follow up. Again, the only valid follow-up uses independent samples and independent platforms. \ --Naomi At 02:38 PM 12/14/2008, Wayne Xu wrote: >Dear Naomi, >I may have a silly question. I read a few papers on microarray >multiple test, I understood what points they were trying to make. >But I still have doubts about it. Since now many journal reviewers >require the FDR for microarray differential expresses genes in >manuscripts, I really want to clear my doubts. > >1). The mathematics model is different from the biology model: >The typical math model to bring up the multiple test issue is >following example: 20 balls in a box with 1 in red and 19 in blue. >The possibility of picking up the red ball from the box each time is >1/20, i.e 0.05. If draw 20 times, the chance is 0.05 multiplied by 20 is 1. >Suppose the red represents false positive, if draw one time the FDR >is 0.05, if 20 times then FDR is 1. People bring this multiple test >issue into microarray data analysis. But in microarray, at least two >aspects are different from this math model: >a). The raw P values are determined by the expression values of >samples, not affected by the total number of genes. So it is >different from above example of 1 out of 20 is 0.05. >b). Pick up a ball and then put it back to the box, you have chance >to pick up the exactly same ball twice or more. But in microarray, >each genes are tested individually at the same time, and each gene >only tested exactly once. >They are obviously different. If this math model is the only reason >that brought up the multiple test issue in microarray, it may be a >misleading (I may be silly, since no one else doubts about multiple >test in microarray?) > >2). Not make biological sense: >Suppose a gene called XYZ has a raw P value of 0.00001 in two group >T test, and it was validated by biological test, e.g. RT-PCR. If the >micoarray chip has 40,000 genes, then by whatever adjustment FDR >method, the adj P-value may be 0.4 or lower or higher. If I use FDR >cutoff 0.1, this XYZ gene has higher FDR and is not in my interest >positive gene list. >OK, now I play a math game, filter gene by variance or other, shrink >the gene list to 5000 (since XYZ gene has low P value, suppose it is >within the 5000). Then the XYZ has low FDR and in my interest >differential gene list. But this is just a math game! >The biological reality is XYZ is positive, this positive is >determined by, for example 4 control samples and 4 treatment >samples, the mean may be big different, and within group variance is >very small. and RT-PCR validated. This reality can not be changed by >whatever number of genes to be tested. The raw P value is close the >biological reality, and it is good to represent the biological >reality. The multiple test here just make you feel happier but not a >biological sense. > >FDR is a very useful term in many biological cases. But it seems >not a good example here for microarray? > >Please help to clear it up. > >Thank you, > >Wayne >-- > > >Naomi Altman wrote: >>Remember that FDR is a rate - i.e. the expected false discovery rate. >>If the set of genes is changeds, FDR will change because the >>comparison set is different. This is NOT the same as a p-value, >>which depends only on the value of the current test statistic. >> >>The same thing happens with FWER, because these methods control the >>probability of making at least one mistake, which clearly depends >>on which set of tests are performed. >> >>--Naomi >> >>At 03:11 PM 12/13/2008, Sean Davis wrote: >>>On Sat, Dec 13, 2008 at 12:36 PM, Wayne Xu <wxu at="" msi.umn.edu=""> wrote: >>> > Hello, >>> > I am not sure this is a right place to ask this question, but it is about >>> > micrarray data analysis: >>> > >>> > In two group t test, the multiple test Q values are depending >>> on the total >>> > number of genes in the test. If I filter the gene list first, >>> for example, I >>> > only use those genes that have1.2 fold changes for T test and >>> multiple test, >>> > this gene list is much smaller than the total gene list, then >>> the multiple >>> > test q values are much smaller. >>> > >>> > Do you think above is a correct way? People who do not do that way may >>> > consider the statistical power may be lost? But how much power >>> lost and how >>> > to calculate the power in this case? >>> >>>No, you cannot filter based on fold change. However, you can filter >>>based on variance or some other measure that does not depend on the >>>two groups being compared. Anything that filters genes based on >>>"knowing" the two groups will lead to a biased test. Remember that >>>filtering removes genes from consideration from further analysis. >>> >>>For further details, there are MANY discussions of this topic in the >>>mailing list. >>> >>> > When people report multiple test Q values, they usually do not >>> mention how >>> > many genes are used in this multiple test. You can get different Q values >>> > (even use the same method, e.g. Benjamin and Holm adjust >>> method) in the same >>> > dataset. Then how can it make sense if the same genes have different Q >>> > values? >>> >>>A good manuscript should describe in detail the preprocessing and >>>filtering steps, the statistical tests used, and the methods for >>>correcting for multiple testing. You are correct that many papers do >>>not do so. >>> >>>As for different q-values in the same dataset using different methods, >>>it is important to note that one should not do an analysis, get a >>>result, and then, based on that result, go back and redo the analysis >>>with different parameters to get a "better" result. It is very >>>important that each step of an analysis (preprocessing, filtering, >>>testing, multiple-testing correction) be justifiable independent of >>>the other steps in order for the results to be interpretable. >>> >>>Sean >>> >>>_______________________________________________ >>>Bioconductor mailing list >>>Bioconductor at stat.math.ethz.ch >>>https://stat.ethz.ch/mailman/listinfo/bioconductor >>>Search the archives: >>>http://news.gmane.org/gmane.science.biology.informatics.conductor >> >>Naomi S. Altman 814-865-3791 (voice) >>Associate Professor >>Dept. of Statistics 814-863-7114 (fax) >>Penn State University 814-865-1348 (Statistics) >>University Park, PA 16802-2111 > >_______________________________________________ >Bioconductor mailing list >Bioconductor at stat.math.ethz.ch >https://stat.ethz.ch/mailman/listinfo/bioconductor >Search the archives: >http://news.gmane.org/gmane.science.biology.informatics.conductor Naomi S. Altman 814-865-3791 (voice) Associate Professor Dept. of Statistics 814-863-7114 (fax) Penn State University 814-865-1348 (Statistics) University Park, PA 16802-2111
ADD REPLY
0
Entering edit mode
Thanks, Naomi, I appreciate this mailing list for providing an opportunity for discussion. I hope more people would be interested in my question too. Wayne -- Naomi Altman wrote: > The ball model does not apply to microarray studies. (And the > probability of drawing the red ball in 20 draws is not 1). > > But FDR does apply to microarray studies, and so does a less discussed > concept, the false nondiscovery rate or FNR. > > Suppose I take 20 independent samples of mouse liver tissue - same > strain, gender ... and hybridize independently to 20 microarrays - any > platform. > Then arbitrarily divide into 2 groups of size 10. If there are 10,000 > genes on the array, you should see 1 gene with p-value .0001or less, > 10 genes with p-value .001 or less, 100 genes with p-value .01 or less > etc. Now suppose you take the 100 genes with the highest degree of > differential expression and do a PCR study with independent samples. > You should still have 1 gene which is significant with p=.01 and 5 > genes which are significant at p=.05. > > The problem is - there is no systematic difference between the > samples. You have detected noise - i.e. chance variation. If you use > the same samples to do your PCR, you may get closer to 100% > "significance" for the selected genes, because the variation that > caused the false detection will still be in the sample unless it was > due only to the hybridization. > > FDR is an estimate of the excess of significant findings, compared to > what is expected by chance. You can reduce FDR greatly by doing > independent follow-up studies (on another microarray or on another > platform such as PCR). You cannot reduce FDR much by reusing the same > samples on a different platform, although you will reduce affects due > to technical variation. > > However, FDR reduces your power to detect differential expression. > This means that you will have higher FNR if you use multiple > comparisons adjustments. Again, if you do independent follow-up > studies, you can reduce FNR. > > The purpose of the FDR computation is to reduce effort wasted on large > gene lists which are mostly reporting noise. But if your genelist is > smaller than you think is reasonable, you may certainly follow up a > larger set of genes and sorting by p-value will give you the most > reasonable set of genes to follow up. Again, > the only valid follow-up uses independent samples and independent > platforms. \ > --Naomi > > At 02:38 PM 12/14/2008, Wayne Xu wrote: >> Dear Naomi, >> I may have a silly question. I read a few papers on microarray >> multiple test, I understood what points they were trying to make. But >> I still have doubts about it. Since now many journal reviewers >> require the FDR for microarray differential expresses genes in >> manuscripts, I really want to clear my doubts. >> >> 1). The mathematics model is different from the biology model: >> The typical math model to bring up the multiple test issue is >> following example: 20 balls in a box with 1 in red and 19 in blue. >> The possibility of picking up the red ball from the box each time is >> 1/20, i.e 0.05. If draw 20 times, the chance is 0.05 multiplied by 20 >> is 1. >> Suppose the red represents false positive, if draw one time the FDR >> is 0.05, if 20 times then FDR is 1. People bring this multiple test >> issue into microarray data analysis. But in microarray, at least two >> aspects are different from this math model: >> a). The raw P values are determined by the expression values of >> samples, not affected by the total number of genes. So it is >> different from above example of 1 out of 20 is 0.05. >> b). Pick up a ball and then put it back to the box, you have chance >> to pick up the exactly same ball twice or more. But in microarray, >> each genes are tested individually at the same time, and each gene >> only tested exactly once. >> They are obviously different. If this math model is the only reason >> that brought up the multiple test issue in microarray, it may be a >> misleading (I may be silly, since no one else doubts about multiple >> test in microarray?) >> >> 2). Not make biological sense: >> Suppose a gene called XYZ has a raw P value of 0.00001 in two group T >> test, and it was validated by biological test, e.g. RT-PCR. If the >> micoarray chip has 40,000 genes, then by whatever adjustment FDR >> method, the adj P-value may be 0.4 or lower or higher. If I use FDR >> cutoff 0.1, this XYZ gene has higher FDR and is not in my interest >> positive gene list. >> OK, now I play a math game, filter gene by variance or other, shrink >> the gene list to 5000 (since XYZ gene has low P value, suppose it is >> within the 5000). Then the XYZ has low FDR and in my interest >> differential gene list. But this is just a math game! >> The biological reality is XYZ is positive, this positive is >> determined by, for example 4 control samples and 4 treatment samples, >> the mean may be big different, and within group variance is very >> small. and RT-PCR validated. This reality can not be changed by >> whatever number of genes to be tested. The raw P value is close the >> biological reality, and it is good to represent the biological >> reality. The multiple test here just make you feel happier but not a >> biological sense. >> >> FDR is a very useful term in many biological cases. But it seems not >> a good example here for microarray? >> >> Please help to clear it up. >> >> Thank you, >> >> Wayne >> -- >> >> >> Naomi Altman wrote: >>> Remember that FDR is a rate - i.e. the expected false discovery rate. >>> If the set of genes is changeds, FDR will change because the >>> comparison set is different. This is NOT the same as a p-value, >>> which depends only on the value of the current test statistic. >>> >>> The same thing happens with FWER, because these methods control the >>> probability of making at least one mistake, which clearly depends on >>> which set of tests are performed. >>> >>> --Naomi >>> >>> At 03:11 PM 12/13/2008, Sean Davis wrote: >>>> On Sat, Dec 13, 2008 at 12:36 PM, Wayne Xu <wxu at="" msi.umn.edu=""> wrote: >>>> > Hello, >>>> > I am not sure this is a right place to ask this question, but it >>>> is about >>>> > micrarray data analysis: >>>> > >>>> > In two group t test, the multiple test Q values are depending on >>>> the total >>>> > number of genes in the test. If I filter the gene list first, for >>>> example, I >>>> > only use those genes that have1.2 fold changes for T test and >>>> multiple test, >>>> > this gene list is much smaller than the total gene list, then the >>>> multiple >>>> > test q values are much smaller. >>>> > >>>> > Do you think above is a correct way? People who do not do that >>>> way may >>>> > consider the statistical power may be lost? But how much power >>>> lost and how >>>> > to calculate the power in this case? >>>> >>>> No, you cannot filter based on fold change. However, you can filter >>>> based on variance or some other measure that does not depend on the >>>> two groups being compared. Anything that filters genes based on >>>> "knowing" the two groups will lead to a biased test. Remember that >>>> filtering removes genes from consideration from further analysis. >>>> >>>> For further details, there are MANY discussions of this topic in the >>>> mailing list. >>>> >>>> > When people report multiple test Q values, they usually do not >>>> mention how >>>> > many genes are used in this multiple test. You can get different >>>> Q values >>>> > (even use the same method, e.g. Benjamin and Holm adjust method) >>>> in the same >>>> > dataset. Then how can it make sense if the same genes have >>>> different Q >>>> > values? >>>> >>>> A good manuscript should describe in detail the preprocessing and >>>> filtering steps, the statistical tests used, and the methods for >>>> correcting for multiple testing. You are correct that many papers do >>>> not do so. >>>> >>>> As for different q-values in the same dataset using different methods, >>>> it is important to note that one should not do an analysis, get a >>>> result, and then, based on that result, go back and redo the analysis >>>> with different parameters to get a "better" result. It is very >>>> important that each step of an analysis (preprocessing, filtering, >>>> testing, multiple-testing correction) be justifiable independent of >>>> the other steps in order for the results to be interpretable. >>>> >>>> Sean >>>> >>>> _______________________________________________ >>>> Bioconductor mailing list >>>> Bioconductor at stat.math.ethz.ch >>>> https://stat.ethz.ch/mailman/listinfo/bioconductor >>>> Search the archives: >>>> http://news.gmane.org/gmane.science.biology.informatics.conductor >>> >>> Naomi S. Altman 814-865-3791 (voice) >>> Associate Professor >>> Dept. of Statistics 814-863-7114 (fax) >>> Penn State University 814-865-1348 (Statistics) >>> University Park, PA 16802-2111 >> >> _______________________________________________ >> Bioconductor mailing list >> Bioconductor at stat.math.ethz.ch >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> Search the archives: >> http://news.gmane.org/gmane.science.biology.informatics.conductor > > Naomi S. Altman 814-865-3791 (voice) > Associate Professor > Dept. of Statistics 814-863-7114 (fax) > Penn State University 814-865-1348 (Statistics) > University Park, PA 16802-2111
ADD REPLY
0
Entering edit mode
I want to reiterate - a q-value is related to FDR. This is the expected percentage of false positives in your gene list if you reject at this p-value or less. It is not the statistical significance of any gene on your gene list. There is however, a Bayesian interpretation that says it is the posterior probability that the gene really differentially expressed. If you are Bayesian, you will always update your posterior with any new information that arises, such as the results of your PCR study. --Naomi At 02:38 PM 12/14/2008, Wayne Xu wrote: >Dear Naomi, >I may have a silly question. I read a few papers on microarray >multiple test, I understood what points they were trying to make. >But I still have doubts about it. Since now many journal reviewers >require the FDR for microarray differential expresses genes in >manuscripts, I really want to clear my doubts. > >1). The mathematics model is different from the biology model: >The typical math model to bring up the multiple test issue is >following example: 20 balls in a box with 1 in red and 19 in blue. >The possibility of picking up the red ball from the box each time is >1/20, i.e 0.05. If draw 20 times, the chance is 0.05 multiplied by 20 is 1. >Suppose the red represents false positive, if draw one time the FDR >is 0.05, if 20 times then FDR is 1. People bring this multiple test >issue into microarray data analysis. But in microarray, at least two >aspects are different from this math model: >a). The raw P values are determined by the expression values of >samples, not affected by the total number of genes. So it is >different from above example of 1 out of 20 is 0.05. >b). Pick up a ball and then put it back to the box, you have chance >to pick up the exactly same ball twice or more. But in microarray, >each genes are tested individually at the same time, and each gene >only tested exactly once. >They are obviously different. If this math model is the only reason >that brought up the multiple test issue in microarray, it may be a >misleading (I may be silly, since no one else doubts about multiple >test in microarray?) > >2). Not make biological sense: >Suppose a gene called XYZ has a raw P value of 0.00001 in two group >T test, and it was validated by biological test, e.g. RT-PCR. If the >micoarray chip has 40,000 genes, then by whatever adjustment FDR >method, the adj P-value may be 0.4 or lower or higher. If I use FDR >cutoff 0.1, this XYZ gene has higher FDR and is not in my interest >positive gene list. >OK, now I play a math game, filter gene by variance or other, shrink >the gene list to 5000 (since XYZ gene has low P value, suppose it is >within the 5000). Then the XYZ has low FDR and in my interest >differential gene list. But this is just a math game! >The biological reality is XYZ is positive, this positive is >determined by, for example 4 control samples and 4 treatment >samples, the mean may be big different, and within group variance is >very small. and RT-PCR validated. This reality can not be changed by >whatever number of genes to be tested. The raw P value is close the >biological reality, and it is good to represent the biological >reality. The multiple test here just make you feel happier but not a >biological sense. > >FDR is a very useful term in many biological cases. But it seems >not a good example here for microarray? > >Please help to clear it up. > >Thank you, > >Wayne >-- > > >Naomi Altman wrote: >>Remember that FDR is a rate - i.e. the expected false discovery rate. >>If the set of genes is changeds, FDR will change because the >>comparison set is different. This is NOT the same as a p-value, >>which depends only on the value of the current test statistic. >> >>The same thing happens with FWER, because these methods control the >>probability of making at least one mistake, which clearly depends >>on which set of tests are performed. >> >>--Naomi >> >>At 03:11 PM 12/13/2008, Sean Davis wrote: >>>On Sat, Dec 13, 2008 at 12:36 PM, Wayne Xu <wxu at="" msi.umn.edu=""> wrote: >>> > Hello, >>> > I am not sure this is a right place to ask this question, but it is about >>> > micrarray data analysis: >>> > >>> > In two group t test, the multiple test Q values are depending >>> on the total >>> > number of genes in the test. If I filter the gene list first, >>> for example, I >>> > only use those genes that have1.2 fold changes for T test and >>> multiple test, >>> > this gene list is much smaller than the total gene list, then >>> the multiple >>> > test q values are much smaller. >>> > >>> > Do you think above is a correct way? People who do not do that way may >>> > consider the statistical power may be lost? But how much power >>> lost and how >>> > to calculate the power in this case? >>> >>>No, you cannot filter based on fold change. However, you can filter >>>based on variance or some other measure that does not depend on the >>>two groups being compared. Anything that filters genes based on >>>"knowing" the two groups will lead to a biased test. Remember that >>>filtering removes genes from consideration from further analysis. >>> >>>For further details, there are MANY discussions of this topic in the >>>mailing list. >>> >>> > When people report multiple test Q values, they usually do not >>> mention how >>> > many genes are used in this multiple test. You can get different Q values >>> > (even use the same method, e.g. Benjamin and Holm adjust >>> method) in the same >>> > dataset. Then how can it make sense if the same genes have different Q >>> > values? >>> >>>A good manuscript should describe in detail the preprocessing and >>>filtering steps, the statistical tests used, and the methods for >>>correcting for multiple testing. You are correct that many papers do >>>not do so. >>> >>>As for different q-values in the same dataset using different methods, >>>it is important to note that one should not do an analysis, get a >>>result, and then, based on that result, go back and redo the analysis >>>with different parameters to get a "better" result. It is very >>>important that each step of an analysis (preprocessing, filtering, >>>testing, multiple-testing correction) be justifiable independent of >>>the other steps in order for the results to be interpretable. >>> >>>Sean >>> >>>_______________________________________________ >>>Bioconductor mailing list >>>Bioconductor at stat.math.ethz.ch >>>https://stat.ethz.ch/mailman/listinfo/bioconductor >>>Search the archives: >>>http://news.gmane.org/gmane.science.biology.informatics.conductor >> >>Naomi S. Altman 814-865-3791 (voice) >>Associate Professor >>Dept. of Statistics 814-863-7114 (fax) >>Penn State University 814-865-1348 (Statistics) >>University Park, PA 16802-2111 > >_______________________________________________ >Bioconductor mailing list >Bioconductor at stat.math.ethz.ch >https://stat.ethz.ch/mailman/listinfo/bioconductor >Search the archives: >http://news.gmane.org/gmane.science.biology.informatics.conductor Naomi S. Altman 814-865-3791 (voice) Associate Professor Dept. of Statistics 814-863-7114 (fax) Penn State University 814-865-1348 (Statistics) University Park, PA 16802-2111
ADD REPLY

Login before adding your answer.

Traffic: 764 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6