I try to examine the effect of "group", controlling the effect of "Mother" (litter). I have 8 different groups and 11 mothers. Both were factored. With a design like this:
dds = DESeqDataSetFromMatrix(countData = cts_filtered, colData = colda_t, design = ~ group + Mother)
I get the familiar error:
Error in checkFullRank(modelMatrix) : the model matrix is not full rank, so the model cannot be fit as specified. One or more variables or interaction terms in the design formula are linear combinations of the others and must be removed. Please read the vignette section 'Model matrix not full rank': vignette('DESeq2')
I thought that meant that one of the variables was redundant (could be inferred from the combination of other variables.) However, I don't think that was the case for my data. Here I print out the samples by group: You can see that a Mother can belong to different groups, and a group contains different Mothers. The only exception is that Mother#3 is always in group TRK, but I thought that wouldn't break the whole thing...
 "UPK" Mother group 22_heart 6 UPK 28_heart 6 UPK 891_heart 9 UPK 901_heart 10 UPK  "TPW" Mother group 23_heart 6 TPW 892_heart 9 TPW 903_heart 10 TPW 914_heart 7 TPW 975_heart 8 TPW 977_heart 8 TPW 981_heart 8 TPW  "UPW" Mother group 24_heart 6 UPW 898_heart 10 UPW 917_heart 7 UPW 925_heart 11 UPW 927_heart 11 UPW 978_heart 8 UPW  "TPK" Mother group 27_heart 6 TPK 924_heart 11 TPK 979_heart 8 TPK 980_heart 8 TPK  "TRW" Mother group 30_heart 2 TRW 37_heart 2 TRW 956_heart 1 TRW 970_heart 5 TRW 971_heart 5 TRW  "URK" Mother group 36_heart 2 URK 63_heart 4 URK 958_heart 1 URK 969_heart 5 URK  "TRK" Mother group 38_heart 2 TRK 74_heart 3 TRK 76_heart 3 TRK 960_heart 1 TRK  "URW" Mother group 59_heart 4 URW 64_heart 4 URW 955_heart 1 URW 959_heart 1 URW
Please let me know why this results in the "checkFullRank(modelMatrix)" error. Thank you!
Mother 3 might be breaking the whole thing...try omitting her and see if it will run. You might still have issues with certain mothers being limited to certain groups.