#### The support.bioconductor.org editor has been updated to markdown! Please see more info at: Tutorial: Updated Support Site Editor

Question: Why NONE of any contrast is significant when overall F-test is significant? using "global" method in decideTests() from limma package
0
11.0 years ago by
Guiyuan Lei90
Guiyuan Lei90 wrote:
Dear all, I am using limma package to identify differential expression. I have 5 contrasts, I used F-statistic to measure significant differential expression, the F-test p-value is adjusted by "fdr" method . I used the command p.adjust(eb$F.p.value, method="fdr") < 0.05 where eb is the object from eBayes(). I got 397 significant differential expression. Then for those 397 significantly differential expressed genes, I want to look at each contrast to check which contrast is significant. I used decideTests(eb, method="global") to classify each t-statistics as up, down or not significant. I found that among genes which are significantly expressed by F-test p-value, that is, the above 397 genes, some have got five '0's from decideTests, which means those genes are not significant for any of its contrast while its F-test is significant. As I understand, if F-test is significant, should at least one of its contrast (t-test) is significant. I doubt that the "global" method used in decideTests() is not properly used for this case? But why? Can any one explain this to me? Many thanks! I attach the code and results as following. library(limma) levels= c('h','h','h','hh','hh','mh','mh','s','s','s','y','y','y','y') dimnames(eset.matrix)[[2]]= levels TS <- factor(levels, levels= c('h','hh','mh','s','y')) design <- model.matrix(~0+TS) colnames(design) <- levels(TS) fit <- lmFit(eset.gcrma, design) #Construct the contrasts mc <- makeContrasts('s-y','h-y','mh-y','hh-h','mh-hh',levels=design) fit2 <- contrasts.fit(fit, mc) eb <- eBayes(fit2) #The adjustment methods using "fdr" modFpvalue <- eb$F.p.value selectedgenesindx <- p.adjust(eb$F.p.value, method="fdr") < 0.05 Sig<-modFpvalue[selectedgenesindx] nsiggenes<-length(Sig) #number of significantly differential expression #decideTests using "global" method results1 <- decideTests(eb, method="global") #Order the F-statistic modF <- eb$F modFordered<-order(modF, decreasing = TRUE) #Get the up or down or not significant for nsiggenes significant (according adjusted p-value of F-test) differential expression updown<-results1[modFordered[1:nsiggenes],] Examples for genes which are significant expression by F-test F-test results from decideTests() eb$F.p.value contrast1 contrast2 contrast3 contrast4 contrast4 5.6E-05 0 0 0 0 0 (this one is not significant for any of its contrast, why five '0's?) 0.00036 0 -1 0 0 0 Best regards, Guiyuan limma • 1.1k views ADD COMMENTlink modified 11.0 years ago by James W. MacDonald49k • written 11.0 years ago by Guiyuan Lei90 Answer: Why NONE of any contrast is significant when overall F-test is significant? usin 0 11.0 years ago by United States James W. MacDonald49k wrote: Hi Guiyan, Guiyuan Lei wrote: > Dear all, > > I am using limma package to identify differential expression. I have 5 > contrasts, I used F-statistic to measure significant differential > expression, the F-test p-value is adjusted by "fdr" method . I used > the command > > p.adjust(eb$F.p.value, method="fdr") < 0.05 > > where eb is the object from eBayes(). I got 397 significant > differential expression. > > Then for those 397 significantly differential expressed genes, I want > to look at each contrast to check which contrast is significant. I > used > > decideTests(eb, method="global") > > to classify each t-statistics as up, down or not significant. > > I found that among genes which are significantly expressed by F-test > p-value, that is, the above 397 genes, some have got five '0's from > decideTests, which means those genes are not significant for any of > its contrast while its F-test is significant. As I understand, if > F-test is significant, should at least one of its contrast (t-test) is > significant. I doubt that the "global" method used in decideTests() is > not properly used for this case? But why? Can any one explain this to > me? Many thanks! I attach the code and results as following. Yes. From ?decideTests: 'method="global"' will treat the entire matrix of t-statistics as a single vector of unrelated tests. So when you did the F-test and adjusted for multiplicity, you adjusted for n tests. Then when you did the t-tests, you did the same, but adjusting for 5n tests. Increasing the number of simultaneous tests five-fold was enough to cause your multiplicity-adjusted p-values to all become insignificant at an alpha of 0.05. You might try method="separate" or "nestedF". Best, Jim > > library(limma) > levels= c('h','h','h','hh','hh','mh','mh','s','s','s','y','y','y','y') > dimnames(eset.matrix)[[2]]= levels > TS <- factor(levels, levels= c('h','hh','mh','s','y')) > design <- model.matrix(~0+TS) > colnames(design) <- levels(TS) > fit <- lmFit(eset.gcrma, design) > > #Construct the contrasts > mc <- makeContrasts('s-y','h-y','mh-y','hh-h','mh-hh',levels=design) > fit2 <- contrasts.fit(fit, mc) > eb <- eBayes(fit2) > > #The adjustment methods using "fdr" > modFpvalue <- eb$F.p.value > selectedgenesindx <- p.adjust(eb$F.p.value, method="fdr") < 0.05 > Sig<-modFpvalue[selectedgenesindx] > nsiggenes<-length(Sig) #number of significantly differential expression > > #decideTests using "global" method > results1 <- decideTests(eb, method="global") > > #Order the F-statistic > modF <- eb$F > modFordered<-order(modF, decreasing = TRUE) > > #Get the up or down or not significant for nsiggenes significant > (according adjusted p-value of F-test) differential expression > updown<-results1[modFordered[1:nsiggenes],] > > Examples for genes which are significant expression by F-test > F-test results from decideTests() > eb$F.p.value contrast1 contrast2 contrast3 contrast4 contrast4 > 5.6E-05 0 0 0 0 0 > (this one is not significant for any of its contrast, why five '0's?) > 0.00036 0 -1 0 0 0 > > > Best regards, > Guiyuan > > _______________________________________________ > Bioconductor mailing list > Bioconductor at stat.math.ethz.ch > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor -- James W. MacDonald, M.S. Biostatistician Affymetrix and cDNA Microarray Core University of Michigan Cancer Center 1500 E. Medical Center Drive 7410 CCGC Ann Arbor MI 48109 734-647-5623
Thank you, Jim. I also tried "separate" and "nestedF". For "separate" method, there are also genes (around 8% of 397 significantly differential genes by F-test) got five '0's. As for "nestedF", every gene got at least one non-zero value. In the post https://stat.ethz.ch/pipermail/bioconductor/2007-April/016698.html Gordon said 'method="separate" and method="nestedF" do quite different things. "separate" controls the FDR on a per-contrast basis only. It does not control the FDR globally across all contrasts. ' Why "separate" method also generated five "0"s (5 contrasts) for significantly differential genes by F-test? Best regards, Guiyuan On Thu, Feb 28, 2008 at 3:54 PM, James W. MacDonald <jmacdon at="" med.umich.edu=""> wrote: > Hi Guiyan, > > > Guiyuan Lei wrote: > > Dear all, > > > > I am using limma package to identify differential expression. I have 5 > > contrasts, I used F-statistic to measure significant differential > > expression, the F-test p-value is adjusted by "fdr" method . I used > > the command > > > > p.adjust(eb$F.p.value, method="fdr") < 0.05 > > > > where eb is the object from eBayes(). I got 397 significant > > differential expression. > > > > Then for those 397 significantly differential expressed genes, I want > > to look at each contrast to check which contrast is significant. I > > used > > > > decideTests(eb, method="global") > > > > to classify each t-statistics as up, down or not significant. > > > > I found that among genes which are significantly expressed by F-test > > p-value, that is, the above 397 genes, some have got five '0's from > > decideTests, which means those genes are not significant for any of > > its contrast while its F-test is significant. As I understand, if > > F-test is significant, should at least one of its contrast (t-test) is > > significant. I doubt that the "global" method used in decideTests() is > > not properly used for this case? But why? Can any one explain this to > > me? Many thanks! I attach the code and results as following. > > Yes. From ?decideTests: > > 'method="global"' will treat the entire matrix of t-statistics as > a single vector of unrelated tests. > > So when you did the F-test and adjusted for multiplicity, you adjusted > for n tests. Then when you did the t-tests, you did the same, but > adjusting for 5n tests. Increasing the number of simultaneous tests > five-fold was enough to cause your multiplicity-adjusted p-values to all > become insignificant at an alpha of 0.05. > > You might try method="separate" or "nestedF". > > Best, > > Jim > > > > > > > > > > library(limma) > > levels= c('h','h','h','hh','hh','mh','mh','s','s','s','y','y','y','y') > > dimnames(eset.matrix)[[2]]= levels > > TS <- factor(levels, levels= c('h','hh','mh','s','y')) > > design <- model.matrix(~0+TS) > > colnames(design) <- levels(TS) > > fit <- lmFit(eset.gcrma, design) > > > > #Construct the contrasts > > mc <- makeContrasts('s-y','h-y','mh-y','hh-h','mh- hh',levels=design) > > fit2 <- contrasts.fit(fit, mc) > > eb <- eBayes(fit2) > > > > #The adjustment methods using "fdr" > > modFpvalue <- eb$F.p.value > > selectedgenesindx <- p.adjust(eb$F.p.value, method="fdr") < 0.05 > > Sig<-modFpvalue[selectedgenesindx] > > nsiggenes<-length(Sig) #number of significantly differential expression > > > > #decideTests using "global" method > > results1 <- decideTests(eb, method="global") > > > > #Order the F-statistic > > modF <- eb$F > > modFordered<-order(modF, decreasing = TRUE) > > > > #Get the up or down or not significant for nsiggenes significant > > (according adjusted p-value of F-test) differential expression > > updown<-results1[modFordered[1:nsiggenes],] > > > > Examples for genes which are significant expression by F-test > > F-test results from decideTests() > > eb\$F.p.value contrast1 contrast2 contrast3 contrast4 contrast4 > > 5.6E-05 0 0 0 0 0 > > (this one is not significant for any of its contrast, why five '0's?) > > 0.00036 0 -1 0 0 0 > > > > > > Best regards, > > Guiyuan > > > > _______________________________________________ > > Bioconductor mailing list > > Bioconductor at stat.math.ethz.ch > > https://stat.ethz.ch/mailman/listinfo/bioconductor > > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor > > -- > James W. MacDonald, M.S. > Biostatistician > Affymetrix and cDNA Microarray Core > University of Michigan Cancer Center > 1500 E. Medical Center Drive > 7410 CCGC > Ann Arbor MI 48109 > 734-647-5623 >