I'm interested in normalizing data from a metagenomic experiment. It's similar to a transcriptomic experiment in that I have counts for several thousand genes across several hundred samples. Two key differences that I'm wary of are (a) this data is much sparser than you would encounter in a transcriptomic experiment and (b) much greater inter sample variability than you would expect in a transcriptomic experiment

Would the mean ratio method (i.e. as called by estimateSizeFactors()) be appropriate for normalization in this case? Are there known core housekeeping genes in a metagenome which can be used as controls for size factor estimation?

You can search here with google for some threads where i talk about DESeq2 for metagenomics / microbiome.

I’m not convinced it’s an optimal method for various reasons but check the threads. I know there was one where i linked to a recent study by Levi Waldron et al.

For size factors, poscounts should be better than default, see the man pages. But it’s also not perfect.