I try to examine the effect of "group", controlling the effect of "Mother" (litter). I have 8 different groups and 11 mothers. Both were factored. With a design like this:

dds = DESeqDataSetFromMatrix(countData = cts_filtered, 
                                colData = colda_t, 
                               design = ~ group + Mother)

I get the familiar error:

Error in checkFullRank(modelMatrix) : 
  the model matrix is not full rank, so the model cannot be fit as specified.
  One or more variables or interaction terms in the design formula are linear
  combinations of the others and must be removed.

  Please read the vignette section 'Model matrix not full rank':

  vignette('DESeq2')

I thought that meant that one of the variables was redundant (could be inferred from the combination of other variables.) However, I don't think that was the case for my data. Here I print out the samples by group: You can see that a Mother can belong to different groups, and a group contains different Mothers. The only exception is that Mother#3 is always in group TRK, but I thought that wouldn't break the whole thing...

[1] "UPK"
          Mother group
22_heart       6   UPK
28_heart       6   UPK
891_heart      9   UPK
901_heart     10   UPK
[1] "TPW"
          Mother group
23_heart       6   TPW
892_heart      9   TPW
903_heart     10   TPW
914_heart      7   TPW
975_heart      8   TPW
977_heart      8   TPW
981_heart      8   TPW
[1] "UPW"
          Mother group
24_heart       6   UPW
898_heart     10   UPW
917_heart      7   UPW
925_heart     11   UPW
927_heart     11   UPW
978_heart      8   UPW
[1] "TPK"
          Mother group
27_heart       6   TPK
924_heart     11   TPK
979_heart      8   TPK
980_heart      8   TPK
[1] "TRW"
          Mother group
30_heart       2   TRW
37_heart       2   TRW
956_heart      1   TRW
970_heart      5   TRW
971_heart      5   TRW
[1] "URK"
          Mother group
36_heart       2   URK
63_heart       4   URK
958_heart      1   URK
969_heart      5   URK
[1] "TRK"
          Mother group
38_heart       2   TRK
74_heart       3   TRK
76_heart       3   TRK
960_heart      1   TRK
[1] "URW"
          Mother group
59_heart       4   URW
64_heart       4   URW
955_heart      1   URW
959_heart      1   URW

Please let me know why this results in the "checkFullRank(modelMatrix)" error. Thank you!

I'd recommend working with a local statistician or someone familiar with linear models in R to choose a design and help with interpretation of results. I have to restrict my time on the support site for software-related issues.