3 months ago by
Walter and Eliza Hall Institute of Medical Research, Melbourne, Australia
If you read the help page for decideTests(), it will answer all your questions.
The default behaviour of decideTests() is to choose genes using an FDR cutoff of 0.05. The default settings are for p.value = 0.05 and adjust = "BH", so the first code option you give in your question is just the same as the default.
When you set lfc = 1.5 you are changing the lfc default but not the p.value or adjust defaults, so you are applying a fold-change cutoff as well as the FDR cutoff. So naturally you must get fewer genes because you are applying two cutoffs at the same time. By setting lfc = 1.5, you are setting the fold change cutoff to be 2^1.5 = 2.83.
If you wanted to set a fold change cutoff of 1.5 without any significance cutoff, then you would use
decideTests(fit2, p = 1, lfc = log2(1.5)).
Regarding which one is better, just read the help page, which says "Although this function enables users to set p-value and lfc cutoffs simultaneously, this combination criterion not usually recommended." We generally recommend a FDR cutoff, which can be made more stringent if you wish using treat().