I have been advised to use filterByExpr after running DESeq2 in order to get rid of a relatively large number of differentially expressed genes which turned out to be expressed in only a few individuals in a few treatment levels.

We have performed RNAseq on a lot of samples of individual flies (~ 20 per condition). Without filterByExpr, I was unable to use group-aware filtering, which I need. Therefore I am now doing :

mm <- model.matrix(~0 + host_treatment_generation, data = sample_data)
mm

keep <- filterByExpr(dds, mm, min.count = 20, min.total.count = 20, large.n = 18, min.prop = 0.7)

dds_keep <- dds[keep,]

thinking this would select genes which have more than 10 counts in at least 70 % of 18 individuals (out of 20, which I would consider a large n).

However, this is not the case, as you can see from the plotted counts of one gene which was detected as being differentially expressed between treatment levels :

there is a maximum of 10 individuals per treatment level which have counts for this gene. Am I doing something wrong ?

You mention "group-aware filtering", and I am guessing that you are expecting filterByExpr to select genes that are expressed a certain number of times in any particular group. But filterByExpr does not do that and, indeed, any group orientated filtering strategy like that would be statistically invalid, either before or after conducting the DE analysis, because it would result in error rate inflation.

filterByExpr instead conducts independent filtering that can be done before the DE analysis without affecting the error rate control. It selects genes that are expressed in a minimum number of samples across the whole experiment, regardless of which groups those samples below to. It uses the design argument only to compute the minimum group size. The filtering conducted by filterByExpr would be exactly the same even if you permuted the group labels or permuted the rows of the design matrix.