I have a RNA-seq experiment with the following design: 

to get genes responding more in medium2 of cellline1 and cellline2 due to treatment, I analyzed both cell lines separately with the following formula:

cell1 = media+treatment+media:treatment

cell2 = media+treatment+media:treatment

Q1. how to fetch genes responding more (more up- or down-regulated), less (less up- or down-regulated) or opposite (up in medium2 but down in medium1) in medium2 vs. medium 1 of individual cell lines?

I want to obtain genes which are responding more (more up- or down-regulted) in medium2(treated) of cellline2 vs. medium2 of cellline1. I used the following design:

I combined cell line and treatment as one factor (cell_treat): e.g., cellline1_ctrl, cellline1_treated

cell2medium2 = media+cell_treat+media:cell_treat

with this formula, I'm getting those genes which are not even significant with their nominal p-values (p<0.05) during pairwise comparisons.

Q2. What am I doing wrong in this design?

Please help to improve my designs!

Thanks

For question 1, your model looks correct, you just need to pull out the interaction term.  It should be evident from running 'resultsNames' which one you need, it'll involve the one with words from both the treatment and media levels.

For question 2, I can interpret it two ways.  When you say "responding" "in medium2(treated) of cellline2 vs. medium2 of cellline1" do you mean

A)  look at the treated vs untreated in the medium 2 of cellline 2, and compare it with the fold-change of treated vs untreated in medium 2 of celline 1, or

B) compare medium 2 treated in cellline 2, and compare it with medium 2 treated in cell-line 1.

I'm not sure what the parenthesised '(treated)' in your question indicates, so it's hard to resolve the exact question.  For the possibility (A), you're going to need a model (saving my typing by using abbreviated factor names) along the lines of ~ (M+ C+T)^3 which will automatically take account of all possible interactions (or M +C + T + C:T + M:C:T,  if you're happy ignoring medium-treatment and cellline-medium interactions).  If you relevel things so that medium2 is the baseline, it will be a bit easier to find the contrast you need, which will be the one corresponding to the C:T term.  It might be conceptually easier to subset the data down to just look at medium2, and then you're back to a two-way interaction, but keeping all the data in will be more powerful.

For possibility (B), it's probably easiest to concatenate the M C and T terms together, and then directly compare the two groups you've specified.

If you want to split the resulting genelists into the different classes (more up, up vs down...), then you're probably best off doing that after the test, by extracting the relevant linear combination of coefficients  - if you  provide some code that allows us to recreate your colData, then it will be easier for us to give specific guidance on this.