Dear Bioconductor users,

I have a problem about the estimation of dispersions in variance stabilizing transformation used by DEseq2 R package .

Specifically, i am working with RNA-seq data (raw counts) and I want to perform WGCNA and feature selection using regularized cox regression modelling (glmnet package). So i want to perform VST transformation that makes them homoscedastic. For estimation of dispersion I used either "parametric" or "local" fittype. Then I constructed the meanSDplots to assess the effect of two alternative fittypes in homoscedasticity. I observed that "local" fittype gives flatter red line (median estimator) but more scattered values (less shrinked standard deviations)  than  "parametric" fittype. Below I have attached the links containing meanSDplots. In your opinion, which is the best tranformation fittype to produce robust data for further analyses?

https://www.dropbox.com/s/tg70hdhf325t4ll/VST_LOCAL_FITTYPE.png?dl=0

https://www.dropbox.com/s/93ehtq6yx2httbs/VST_PARAMETRIC_FITTYPE.png?dl=0

Thank you for your time in advance!!

Sincerely,

Panagiotis Mokos

hi Panagiotis,

Thanks for posting here.

Both of those plots look fine to me. I wouldn't worry too much about slight movements in the red line and focus on the blue density. Basically what you want to see is that the density is relatively flat (not that the left side or right side is much higher). See here for examples of data that is not variance stabilized:

http://www.bioconductor.org/help/workflows/rnaseqGene/#the-rlog-and-variance-stabilizing-transformations