I am trying to understand exactly what DESeq2's resultsNames outputs. The documentation says:

resultsNames returns the names of the estimated effects (coefficents) of the model

For a simple example, it makes sense to me:

design(dds) <- ~ group
> resultsNames(dds)
[1] "Intercept" "groupX"    "groupY"   

For something more complicated, I am confused:

design(dds) <- ~ group + condition + group:condition
> resultsNames(dds)
[1] "Intercept"         "group_Y_vs_X"      "condition_B_vs_A"  "groupY.conditionB"

I am not clear how both outputs are compatible with each other. I understand they should not be identical, but they seem to be very different to me. Is there a different way of phrasing it?

If you create the model.matrix manually from the design you specify, ie. something like this:

mm <- model.matrix(~ group + condition + group:condition,
                   as.data.frame(colData(dds)))

The names you get from resultNames(dds) should match 1:1 the columns (coefficients) of your design. I'm not sure if this level of detail adds enough clarity for you. If not, please ask for further (specific) clarification.

We are adding more explanation to the vignette for just this question, but I'll explain here as well. We mention in the vignette that most designs use an "expanded model matrix" (the first resultsNames you show), but for the case of a design with interactions and when all factors have only two levels, we use the standard model matrix produced by model.matrix() in base R. The reason for this switch from the default, is so that it is easy to test if the interaction effect is significant, by testing a single term: for example "groupY.conditionB".

For more explanation of the purpose of expanded model matrices, check first section 4.5 of the vignette or the section of the Methods of our publication which discusses the expanded model matrix.

Currently in vignette section 4.5 we have: "Expanded model matrices are not used without the log2 fold change prior or in the case of designs with 2 level factors and an interaction term." and in ?nbinomWaldTest we have: "nbinomWaldTest will by default use expanded model matrices, as described in the modelMatrixType argument, unless this argument is used to override the default behavior or unless the design contains 2 level factors and an interaction term". But as I said, we are writing more explanation for this switch in defaults.