I am trying to plot a sample heatmap for RNA-seq data. I used rlog to transform count data since I have only 6 samples.
The sample heatmap did not have a good cluster so I tried transforming the data with VST.
I then tried to plot the standard deviation against the mean of transformed counts by rlog and VST, both figures have a log scaled axis (see figures below, left is SD & mean of rlog count, and right is for VST counts)
**Could you please explain why the SD of the rlog counts looks that way and is this abnormal?**
**I was expecting that after the transformation, the gene with higher mean values will have SD scattered in the same range (similar to the right figure) meaning that the data becomes more homoscedastic, is this a correct assumption?**
previously, I used rlog on data from another cell line (the SDvMean plot looks okay for that one), so I prefer to use rlog for consistency, when possible.