I have grown diseased cells (with 3 different genotypes, type A, B, and C) and healthy cells (WT) and put them in two different culture dimensionalities, 2D and 3D culture. I then have 8 groups: 2D healthy, 3D healthy, 2D diseased (of genotype A, B, C) and 3D diseased (of genotype A, B, C). Additionally, the samples are derived from two experimental iterations (E1 and E2), performed at different times.

The experimental design then looks like this:

> coldata
   SampleName Experiment condition genotype
1   E1_2D_A_1         E1        2D        A
2   E1_2D_A_2         E1        2D        A
3   E1_3D_A_1         E1        3D        A
4   E1_3D_A_2         E1        3D        A
5   E2_2D_A_1         E2        2D        A
6   E2_2D_A_2         E2        2D        A
7   E2_3D_A_1         E2        3D        A
8   E2_3D_A_2         E2        3D        A
9   E1_2D_B_1         E1        2D        B
10  E1_3D_B_1         E1        3D        B
11  E1_3D_B_2         E1        3D        B
12  E2_2D_B_1         E2        2D        B
13  E2_2D_B_2         E2        2D        B
14  E2_3D_B_1         E2        3D        B
15  E2_3D_B_2         E2        3D        B
16  E1_2D_C_1         E1        2D        C
17  E1_2D_C_1         E1        2D        C
18  E1_3D_C_1         E1        3D        C
19  E1_3D_C_2         E1        3D        C
20  E1_3D_C_1         E1        3D        C
21  E1_3D_C_2         E1        3D        C
22  E2_2D_C_1         E2        2D        C
23  E2_2D_C_1         E2        2D        C
24  E2_2D_C_2         E2        2D        C
25  E2_3D_C_1         E2        3D        C
26  E2_3D_C_2         E2        3D        C
27  E2_3D_C_1         E2        3D        C
28  E2_3D_C_2         E2        3D        C
29 E1_2D_WT_1         E1        2D       WT
30 E1_3D_WT_1         E1        3D       WT
31 E1_3D_WT_2         E1        3D       WT
32 E2_2D_WT_1         E2        2D       WT
33 E2_2D_WT_2         E2        2D       WT
34 E2_2D_WT_1         E2        2D       WT
35 E2_2D_WT_2         E2        2D       WT
36 E2_3D_WT_1         E2        3D       WT
37 E2_3D_WT_2         E2        3D       WT
38 E2_3D_WT_3         E2        3D       WT
39 E2_3D_WT_1         E2        3D       WT
40 E2_3D_WT_2         E2        3D       WT

First I did comparisons within each culture condition by grouping condition and genotype and using lfcshrink/ashr. For instance for genotype A:

dds <- DESeqDataSetFromMatrix(countData = countdata,
                              colData = coldata,
                              design = ~ Experiment + condition + genotype)

dds$group <- factor(paste0(dds$genotype, dds$condition))
design(dds) <- ~ group
dds <- DESeq(dds)

#2D contrast
A_2D_v_WT_2D_lfcShrink <- lfcShrink(dds, contrast=c("group","A2D","WT2D"), type = "ashr", alpha = 0.05)
#3D contrast
A_2D_v_WT_2D_lfcShrink <- lfcShrink(dds, contrast=c("group","A3D","WT3D"), type = "ashr", alpha = 0.05)

I then ran GSEA observed that some disease-related pathways in 3D have much smaller p-values than the 2D comparison. I looked deeper into why this might be by looking at genes' log2FCs that represent those pathways. Turns out that some of the log2FCs are larger in the 3D comparison than the 2D one. The question is now: for a given gene, how do I determine if the log2FC in the 3D comparison is statistically larger than the log2FC in the 2D comparison?

After reading the vignette and exploring other people's questions on support sites, I think it is possible to do this in DESeq2, namely to compare two contrasts. In my case, between 3D and 2D within the same genotype, normalized to their respective WT groups by use of a design like design = ~ Experiment + condition + genotype + condition:genotype

However, I would like verification that this is the right way to go to answer the question I have. If so, I need help on how to go from the design into making the contrast successfully since I am a novice. I have tried to follow the "Interactions" section in the 10.13.2020 DESeq2 guide and ?results, but unsure how to confirm the function it is making the comparisons I think it is making. Explicitly, how to make sure the contrast is using the correct reference level (either WT in 3D, or WT in 2D for A in 3D and A in 2D, respectively).

Thank you in advance for your time and help.

Yes, the way to ask if a LFC (say associated with condition) is larger in one group than another (say stratified by a variable group) is: ~group + condition + group:condition. If the interaction effect size is ~0 this means the LFC are similar, if it is larger than zero this means that the LFC is larger in the non-reference group, etc.

If you want to check your intuition, look at genes with a significant interaction term and then look at the plotCounts for those genes.

Re: what reference level is used, this is based on levels(dds$condition) which the user has control over (see vignette note on factor levels), or you can also just look at resultsNames(dds) to see what the reference levels were set to.