While analyzing RNA-seq data for differentially expressed genes, I was wondering if it would be appropriate to use the "ratio of small p values vs mean normalized count" diagnostic plot, to decide upon a cutoff of counts to consider as "minimum counts" to shortlist genes for further analysis.

Additionally, I was not able to understand, exactly what ratio is computed on the Y axis as "Ratio of small p-values", I understand that a certain small p-value cutoff (like 0.05?) is considered, and, counts are averaged for all rows below that p-value cutoff, but what exactly is the ratio computed from?

Thanks

This is what DESeq2 does internally (finds a cutoff to reduce the multiple testing burden, then applies BH procedure to the gene passing the filter), see this section:

https://bioconductor.org/packages/release/bioc/vignettes/DESeq2/inst/doc/DESeq2.html#independent-filtering-and-multiple-testing

You may also be interested in the more rigorous method, IHW, which can be easily called from DESeq2:

https://bioconductor.org/packages/release/bioc/vignettes/DESeq2/inst/doc/DESeq2.html#independent-hypothesis-weighting