I am trying to use limma-voom to determine the effect of a condition between males and females. Even after reading through the users guide and several similar posts online I am still confused and would very much appreciate some guidance.

I have paired samples, and my data looks as such (following the edgeR user's guide):

As of right now, here is how I set up my design matrix and collect the results:

design = ~ sex + sex:nested + sex:condition

v = voom(dge, design, plot=FALSE)

fit = lmFit(v, design)

#construct contrast matrix

p <- ncol(fit)
cont <- rep(0, p)
cont[p] <- 1
cont[p-1] <- -1

fit2 = contrasts.fit(fit, cont)

fit2 = eBayes(fit2)

results = decideTests(fit2)

> summary(results)
   [,1]
-1     0
0  21457
1      0

It seems strange that I'm getting nothing, leading me to believe that I'm testing the contrast incorrectly. Did I fit the contrast correctly, or is it a problem with how I set up the interaction terms in my design? I'm pretty confused, so any help would be greatly appreciated.

R version 3.3.1 (2016-06-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS release 6.8 (Final)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] edgeR_3.16.5  limma_3.30.10

loaded via a namespace (and not attached):
[1] grid_3.3.1      locfit_1.5-9.1  lattice_0.20-33

For starters, you should define the design matrix with model.matrix. As far as I know, voom does not support formula input. You should have done something like this already:

design <- model.matrix(~ sex + sex:nested + sex:condition)

Secondly, you should make sure nested is a factor and not a covariate. Otherwise you'd be effectively fitting a line to the samples from each sex, which is probably not what you intended.

If you still don't get any DE genes, have a look at a MDS plot to see whether you have any outlier samples that might be inflating the variance. If so, these samples can be downweighted with voomWithQualityWeights. You could also turn on robust=TRUE in eBayes to protect against outlier expression for a small number of genes.

If you still don't get any DE genes... then maybe you just don't have any differences in the treatment response between sexes in your data set. The contrast terms seem correct, though you don't show the column names of the design matrix in your post.