Why NONE of any contrast is significant when overall F-test is significant? using "global" method in decideTests() from limma package
1
0
Entering edit mode
Guiyuan Lei ▴ 90
@guiyuan-lei-2506
Last seen 9.6 years ago
Dear all, I am using limma package to identify differential expression. I have 5 contrasts, I used F-statistic to measure significant differential expression, the F-test p-value is adjusted by "fdr" method . I used the command p.adjust(eb$F.p.value, method="fdr") < 0.05 where eb is the object from eBayes(). I got 397 significant differential expression. Then for those 397 significantly differential expressed genes, I want to look at each contrast to check which contrast is significant. I used decideTests(eb, method="global") to classify each t-statistics as up, down or not significant. I found that among genes which are significantly expressed by F-test p-value, that is, the above 397 genes, some have got five '0's from decideTests, which means those genes are not significant for any of its contrast while its F-test is significant. As I understand, if F-test is significant, should at least one of its contrast (t-test) is significant. I doubt that the "global" method used in decideTests() is not properly used for this case? But why? Can any one explain this to me? Many thanks! I attach the code and results as following. library(limma) levels= c('h','h','h','hh','hh','mh','mh','s','s','s','y','y','y','y') dimnames(eset.matrix)[[2]]= levels TS <- factor(levels, levels= c('h','hh','mh','s','y')) design <- model.matrix(~0+TS) colnames(design) <- levels(TS) fit <- lmFit(eset.gcrma, design) #Construct the contrasts mc <- makeContrasts('s-y','h-y','mh-y','hh-h','mh-hh',levels=design) fit2 <- contrasts.fit(fit, mc) eb <- eBayes(fit2) #The adjustment methods using "fdr" modFpvalue <- eb$F.p.value selectedgenesindx <- p.adjust(eb$F.p.value, method="fdr") < 0.05 Sig<-modFpvalue[selectedgenesindx] nsiggenes<-length(Sig) #number of significantly differential expression #decideTests using "global" method results1 <- decideTests(eb, method="global") #Order the F-statistic modF <- eb$F modFordered<-order(modF, decreasing = TRUE) #Get the up or down or not significant for nsiggenes significant (according adjusted p-value of F-test) differential expression updown<-results1[modFordered[1:nsiggenes],] Examples for genes which are significant expression by F-test F-test results from decideTests() eb$F.p.value contrast1 contrast2 contrast3 contrast4 contrast4 5.6E-05 0 0 0 0 0 (this one is not significant for any of its contrast, why five '0's?) 0.00036 0 -1 0 0 0 Best regards, Guiyuan
limma limma • 1.8k views
ADD COMMENT
0
Entering edit mode
@james-w-macdonald-5106
Last seen 11 minutes ago
United States
Hi Guiyan, Guiyuan Lei wrote: > Dear all, > > I am using limma package to identify differential expression. I have 5 > contrasts, I used F-statistic to measure significant differential > expression, the F-test p-value is adjusted by "fdr" method . I used > the command > > p.adjust(eb$F.p.value, method="fdr") < 0.05 > > where eb is the object from eBayes(). I got 397 significant > differential expression. > > Then for those 397 significantly differential expressed genes, I want > to look at each contrast to check which contrast is significant. I > used > > decideTests(eb, method="global") > > to classify each t-statistics as up, down or not significant. > > I found that among genes which are significantly expressed by F-test > p-value, that is, the above 397 genes, some have got five '0's from > decideTests, which means those genes are not significant for any of > its contrast while its F-test is significant. As I understand, if > F-test is significant, should at least one of its contrast (t-test) is > significant. I doubt that the "global" method used in decideTests() is > not properly used for this case? But why? Can any one explain this to > me? Many thanks! I attach the code and results as following. Yes. From ?decideTests: 'method="global"' will treat the entire matrix of t-statistics as a single vector of unrelated tests. So when you did the F-test and adjusted for multiplicity, you adjusted for n tests. Then when you did the t-tests, you did the same, but adjusting for 5n tests. Increasing the number of simultaneous tests five-fold was enough to cause your multiplicity-adjusted p-values to all become insignificant at an alpha of 0.05. You might try method="separate" or "nestedF". Best, Jim > > library(limma) > levels= c('h','h','h','hh','hh','mh','mh','s','s','s','y','y','y','y') > dimnames(eset.matrix)[[2]]= levels > TS <- factor(levels, levels= c('h','hh','mh','s','y')) > design <- model.matrix(~0+TS) > colnames(design) <- levels(TS) > fit <- lmFit(eset.gcrma, design) > > #Construct the contrasts > mc <- makeContrasts('s-y','h-y','mh-y','hh-h','mh-hh',levels=design) > fit2 <- contrasts.fit(fit, mc) > eb <- eBayes(fit2) > > #The adjustment methods using "fdr" > modFpvalue <- eb$F.p.value > selectedgenesindx <- p.adjust(eb$F.p.value, method="fdr") < 0.05 > Sig<-modFpvalue[selectedgenesindx] > nsiggenes<-length(Sig) #number of significantly differential expression > > #decideTests using "global" method > results1 <- decideTests(eb, method="global") > > #Order the F-statistic > modF <- eb$F > modFordered<-order(modF, decreasing = TRUE) > > #Get the up or down or not significant for nsiggenes significant > (according adjusted p-value of F-test) differential expression > updown<-results1[modFordered[1:nsiggenes],] > > Examples for genes which are significant expression by F-test > F-test results from decideTests() > eb$F.p.value contrast1 contrast2 contrast3 contrast4 contrast4 > 5.6E-05 0 0 0 0 0 > (this one is not significant for any of its contrast, why five '0's?) > 0.00036 0 -1 0 0 0 > > > Best regards, > Guiyuan > > _______________________________________________ > Bioconductor mailing list > Bioconductor at stat.math.ethz.ch > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor -- James W. MacDonald, M.S. Biostatistician Affymetrix and cDNA Microarray Core University of Michigan Cancer Center 1500 E. Medical Center Drive 7410 CCGC Ann Arbor MI 48109 734-647-5623
ADD COMMENT
0
Entering edit mode
Thank you, Jim. I also tried "separate" and "nestedF". For "separate" method, there are also genes (around 8% of 397 significantly differential genes by F-test) got five '0's. As for "nestedF", every gene got at least one non-zero value. In the post https://stat.ethz.ch/pipermail/bioconductor/2007-April/016698.html Gordon said 'method="separate" and method="nestedF" do quite different things. "separate" controls the FDR on a per-contrast basis only. It does not control the FDR globally across all contrasts. ' Why "separate" method also generated five "0"s (5 contrasts) for significantly differential genes by F-test? Best regards, Guiyuan On Thu, Feb 28, 2008 at 3:54 PM, James W. MacDonald <jmacdon at="" med.umich.edu=""> wrote: > Hi Guiyan, > > > Guiyuan Lei wrote: > > Dear all, > > > > I am using limma package to identify differential expression. I have 5 > > contrasts, I used F-statistic to measure significant differential > > expression, the F-test p-value is adjusted by "fdr" method . I used > > the command > > > > p.adjust(eb$F.p.value, method="fdr") < 0.05 > > > > where eb is the object from eBayes(). I got 397 significant > > differential expression. > > > > Then for those 397 significantly differential expressed genes, I want > > to look at each contrast to check which contrast is significant. I > > used > > > > decideTests(eb, method="global") > > > > to classify each t-statistics as up, down or not significant. > > > > I found that among genes which are significantly expressed by F-test > > p-value, that is, the above 397 genes, some have got five '0's from > > decideTests, which means those genes are not significant for any of > > its contrast while its F-test is significant. As I understand, if > > F-test is significant, should at least one of its contrast (t-test) is > > significant. I doubt that the "global" method used in decideTests() is > > not properly used for this case? But why? Can any one explain this to > > me? Many thanks! I attach the code and results as following. > > Yes. From ?decideTests: > > 'method="global"' will treat the entire matrix of t-statistics as > a single vector of unrelated tests. > > So when you did the F-test and adjusted for multiplicity, you adjusted > for n tests. Then when you did the t-tests, you did the same, but > adjusting for 5n tests. Increasing the number of simultaneous tests > five-fold was enough to cause your multiplicity-adjusted p-values to all > become insignificant at an alpha of 0.05. > > You might try method="separate" or "nestedF". > > Best, > > Jim > > > > > > > > > > library(limma) > > levels= c('h','h','h','hh','hh','mh','mh','s','s','s','y','y','y','y') > > dimnames(eset.matrix)[[2]]= levels > > TS <- factor(levels, levels= c('h','hh','mh','s','y')) > > design <- model.matrix(~0+TS) > > colnames(design) <- levels(TS) > > fit <- lmFit(eset.gcrma, design) > > > > #Construct the contrasts > > mc <- makeContrasts('s-y','h-y','mh-y','hh-h','mh- hh',levels=design) > > fit2 <- contrasts.fit(fit, mc) > > eb <- eBayes(fit2) > > > > #The adjustment methods using "fdr" > > modFpvalue <- eb$F.p.value > > selectedgenesindx <- p.adjust(eb$F.p.value, method="fdr") < 0.05 > > Sig<-modFpvalue[selectedgenesindx] > > nsiggenes<-length(Sig) #number of significantly differential expression > > > > #decideTests using "global" method > > results1 <- decideTests(eb, method="global") > > > > #Order the F-statistic > > modF <- eb$F > > modFordered<-order(modF, decreasing = TRUE) > > > > #Get the up or down or not significant for nsiggenes significant > > (according adjusted p-value of F-test) differential expression > > updown<-results1[modFordered[1:nsiggenes],] > > > > Examples for genes which are significant expression by F-test > > F-test results from decideTests() > > eb$F.p.value contrast1 contrast2 contrast3 contrast4 contrast4 > > 5.6E-05 0 0 0 0 0 > > (this one is not significant for any of its contrast, why five '0's?) > > 0.00036 0 -1 0 0 0 > > > > > > Best regards, > > Guiyuan > > > > _______________________________________________ > > Bioconductor mailing list > > Bioconductor at stat.math.ethz.ch > > https://stat.ethz.ch/mailman/listinfo/bioconductor > > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor > > -- > James W. MacDonald, M.S. > Biostatistician > Affymetrix and cDNA Microarray Core > University of Michigan Cancer Center > 1500 E. Medical Center Drive > 7410 CCGC > Ann Arbor MI 48109 > 734-647-5623 >
ADD REPLY

Login before adding your answer.

Traffic: 888 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6