-After i downloaded the TCGA raw gene counts. I created a DESeq object using Design=~1

-To visualize the normalized effect I ran this code lon <- log2(counts(dds, normalized=TRUE) + 1)

-I also ran this code r<-rlog(dds)

clustering both matrices results in different outcome.

Any insight???

Yes, rlog should give different clustering results from a simple log2 transformation. The effect of rlog in clustering is to give less weight to genes with small counts, since small counts are more affected by counting noise and thus carry less information about the biology of the samples. If your samples' clustering pattern is strongly influenced by genes with small counts, then rlog, as well as any other kind of variance-stabilizing transformation, will most definitely change that clustering pattern. This change is usually an improvement.

Yes, of course: this is exactly the point of using these transformations vs +1 smoothing followed by log2 transformation.

This is covered in the entirety of section 2 of the DESeq2 vignette, which would be a worthwhile read.

Thank you so much...I found out that rlog should not be used in zero infested matrix with large amount of samples.