Just as a an additional side note: commonly, the blocking factor is the second
coefficient, and the the coefficient representing the fold change of interest
is put last.
However, this is just as cosmetic in limma, since you specify explicitly which coefficient to extract, while e.g. in DESeq2 the last coefficient is tested by default. I.e.
design<-model.matrix(~Sex + Sample)
Additionally, you can also inspect the p-values for the blocking
factor "Sex", if you do not have a lot of rejections there, you
might even leave it out entirely and do a simple two groups