We would like to remove counts that are >2 SD away from the mean within each group, as was directed by our statistician.

After performing DESeq Differential Expression Analysis on our raw counts and obtaining normalized count values, we found some genes that contained extreme outliers within their normalized count values.

First, is this a an appropriate task for differential expression analysis, or does it violate any rules within DEG analysis?**

My data has two treatment groups, each with 7-12 subjects.

If I remove the outliers ( | NormalizedCounts | > 2 SD from group mean of counts ), I am not certain how to perform differential expression analysis on the normalized counts.

I read the documentation on the DESeq(dds, minReplicatesForReplace = Inf)

function, but am unclear if that would remove the outlier filter built into DESeq, or if there are other parameters I can set it to customize the outlier threshold.

If I can't customize the outlier threshold in DESeq when using my raw count values as input, is there a way to run analysis on normalized counts (after outlier removal)?

I wouldn't remove outliers based on SD.

We have a formal outlier procedure in DESeq2 which has already been tested during the development and 2014 publication, which I would recommend instead if you are worried about the effect of outliers.

Note that setting minReplicatesForReplace = Inf turns off outlier replacement, but it will still filter genes (set p-values to NA) which contain outliers.

There are parameters for the outlier threshold, see cooksCutoff argument in ?results.