Hi,

When applying vst or rlog transformations to rna-seq data to latter on visualize it in a heatmap with dendrograms, is vst/rlog applied to raw counts or instead to normalized counts when we do assay(rlog(ddsHTSeq, blind=F)) ?

By normalized counts I mean corrected with scaling factor calculated with DEseq() command.

In case it's applied to raw counts, shouldn't we correct vst/rlog values someway for sequencing depth and average gene expression across samples before plotting heatmap and dendrograms?

Best

Have a look at the manual pages of these functions. The first sentence of that for varianceStabilizingTransformation says "This function calculates a variance stabilizing transformation (VST) from the fitted dispersion-mean relation(s) and then transforms the count data (normalized by division by the size factors or normalization factors)." For rlog, it says "This function transforms the count data to the log2 scale in a way which minimizes differences between samples for rows with small counts, and which normalizes with respect to library size."

Do try to read the documentation and a little bit about the underlying methods, you'll find that you'll be more efficient and have much more fun with the software.