I ran into some complications while analyzing/normalizing my data. It would be kind of you if you could help me.

We have ChIP-seq data for 5 histones from treated and untreated mice. Each histone has two biological replicates and one technical replicate of only one biological replicate. Before performing any differential testing, I wanted to check if the normalization with the size factors from DESeq2 will improve my data or show some possible experiment errors etc. Also, I am intended to use normalized counts (by size factors) outside the DESeq to check for differential analysis and also VST and rlog transformed data as input for PEER. 

Raw counts for histone K4me3:

Square root of counts from histone K4me3:



After normalization with the sizeFactors:
cds=estimateSizeFactors(dds)
o=counts(cds,normalized=TRUE)
First pic are the normalized counts and the second is the square root on the normalized counts.


VST and rlog transformations:


It seems that something strange is happening there. Also the same plots for other data show some kind of strange curves after the normalization.

Normalization in ChIP-seq samples is a complicated question. The correct normalization to use depends on how you did your counts, how you did your experiment, what questions you want to ask, the nature of the histone marks being studied, and the consistency of the antibodies used to IP. The csaw package's user guide has an entire section on ChIP-seq normalization, which you should read in its entirety before continuing. But broadly speaking, histone mark data generally has a bimodal distribution low-abundance peak consisting of counts for unmarked regions, and a higher-abundance peak consisting of counts in regions with the histone mark, and the relationship between the two peaks depends on lots of factors. Because of this, methods for estimating size factors in RNA-seq may not be appropriate for ChIP-Seq. (Also, it may be difficult to normalize different histone marks together, depending on how similar their binding profiles are.)

Once you have chosen suitable size factors, though, both the VST and rlog transformations are probably reasonable, as Michael Love has explained.

I'm not sure exactly what you are referring to as strange, but here's my thoughts:

VST and rlog are similar to log2(K_ij / s_j). You can read the DESeq2 paper to see what kind of model the rlog is built on. It works well for many RNA-seq experiments, but we didn't try it on ChIP-seq experiments. The VST is a more general transformation, which estimates overdispersion and then applies the transformation which should approximately stabilize the variance for the null "genes". You can read the DESeq paper for more details on this.

From looking at these plots, I would go with VST or log2(normalized counts + pseudocount), but it's in general hard to say for a given experiment, what is noise and what is signal, especially without replicates.

You could also try quantile normalization if you want the distributions to align.