Dear Bioconductor community,

In a couple of my comparisons using the mroast gene set test I received some confusing results (as shown below). Notably, I understand that there are no significantly differentially expressed genes in either (up or down) or both directions (both prior to and following Benjamini-Hochberg adjustment of the p-values). However, I'm wondering why the direction of the M2a set is "Up" when the proportion of genes down in the M2a set is greater than the proportion of genes up in the M2a set. Am I missing something important regarding the interpretation of the results?

Thank you for your time and looking forward to your response, 

SB

mroast(data, gs.indices, design, contrast = makeContrasts(Condition-Ctrl,levels=design), 
       geneid = NULL, set.statistic = "mean", gene.weights = NULL, 
       var.prior = NULL, df.prior = NULL,
       nrot = 999, approx.zscore = TRUE, adjust.method = "BH",
       midp = TRUE, sort = "none")

        NGenes    PropDown     PropUp           Direction  PValue   FDR          PValue.Mixed    FDR.Mixed
M1.Set      76    0.13157895   0.06578947    Down       0.312     0.623         0.52                   0.9795
M2a.Set     53   0.05660377   0.03773585     Up           0.843     0.843         0.98                   0.9800 

The proportion of genes that go in a certain direction isn't necessarily informative. The statistic you use when you choose 'mean' is the mean of the t-statistics for each set of genes. You might have more genes that go down (say 5.7%) than go up (say 3.8%), but that doesn't tell you anything about the t-statistics for those genes. If the t-statistics for the genes that go up are much larger than those that go down, then the mean of those statistics will be positive, and thus the 'Direction' of the gene set will be 'Up', and the reported p-value and FDR will pertain to that direction as well.

There is no conflict here. The roast p-values are large, so the two sets are not even remotely statistically significant. Also the proportions of Up and Down genes are very small, as would be expected for non-significant sets. So the two pieces of information are in agreement. 

As James has pointed out, the Direction of change is computed from the set.statistic that you set in the function call, not from PropUp and PropDown. For sets with no signal the direction of change will obviously be random, so you shouldn't be interpreting it anyway.