Entering edit mode
I would like to have a systematic procedure, when choosing model of mean-dispersion trend in DESeq2 when using estimateDispersions() function. I know that 4 different models are available with strong impact on trend estimation. However, in many cases, "goodness-of-fit" cannot be achieve only with graphical analysis but we need something more systematic. So, my question is: "Does it exist any procedure to compare models when choosing different fitType in estimateDispersions()?". I am not aware of some likelihood or deviance returned by DESeq2 in order to proceed to model comparison.
Any insights will be appreciated.
Loïc
Thanks Michael Love ! Ok so let's summarize for my own understanding. If I want to proceed to some model comparison of the trend, fitting the models by my own only considering genes with average counts > 10 should be reasonable ? For giving you a little bit of context, in the lab where I work, people are interested in systematic model comparison. Since I am only a statistician I try to find a reasonable approach to do so.
Yes, if you see that part in the paper, we argue that when average count is less than 1/alpha then you have little information for estimation of dispersion. alpha could be .1 or even .01 so if you really wanted to be safe, filter at a higher average count, like 100.
I will go through the paper one more time just to be sure I got the point. Thanks for your help !