My RNA-seq data is highly nested and unbalanced: plants within genotype within family with different numbers of plants within each genotype and different numbers of genotypes within each family. When I did the analysis using contrasts, I simply averaged the genotype means within family, which was the equivalent of using a sample weight: 1/(n_P.g * n_g.F) where n_P.g is the number of plants (samples) within a genotype and and n_g.F is the number of genotypes in the family.
Now I want to use a continuous predictor (one value per plant) instead of contrasts. However, I still think that I need to weighted analysis to deal with the nesting and imbalance.
1) In LIMMA can I use the sample quality weights to impose this weighting? If so, do I also use the voom weights?
2) Is there a way to do this in DESeq2 with the GLRT?