My RNA-seq data is highly nested and unbalanced: plants within genotype within family with different numbers of plants within each genotype and different numbers of genotypes within each family. When I did the analysis using contrasts, I simply averaged the genotype means within family, which was the equivalent of using a sample weight: 1/(n_P.g * n_g.F) where n_P.g is the number of plants (samples) within a genotype and and n_g.F is the number of genotypes in the family.

Now I want to use a continuous predictor (one value per plant) instead of contrasts. However, I still think that I need to weighted analysis to deal with the nesting and imbalance.

1) In LIMMA can I use the sample quality weights to impose this weighting? If so, do I also use the voom weights?

2) Is there a way to do this in DESeq2 with the GLRT?

--Naomi

We don't have contrasts implemented for the LRT in DESeq2, only for the Wald test (in results(), the contrast argument can be a numeric vector and will generate a new Wald statistic).

Naomi, I don't really follow your experiment or why a continous variable causes a problem, but I think you already know how to use weights, contrasts and continuous covariates in limma. If necessary you can make the sample weights a function of a continuous variable in limma using the var.design argument of arrayWeights or voomLmFit. I'm not sure what you have in mind, but I don't think it is correct to set weight equal to the number of repeated measures samples being averaged, if that's what you are thinking of, because the higher level variance is not being averaged.