Are the assumptions for normalization violated when comparing experimental conditions with different ploidy?
Strictly speaking, the answer is probably yes, due to gene dosage effects. While the increase in expression might not be exactly equal to the increase in dosage (e.g., due to negative feedback loops), I would find it hard to believe that most genes are not affected by throwing another (active) copy of the genome into the nucleus. This should result in upregulation of the majority of genes, and if most genes are DE, standard normalization methods like TMM won't work. You would need to add spike-in RNA instead, to control for changes in input quantity - by its nature, sequencing will only capture the composition of a population of (c)DNA molecules.
Does it make sense to perform differential expression analysis between them with RNA-seq?
A DE analysis may still be interesting in this context. Let us assume that the expression of most genes are affected by the change in dosage in a straightforward and uninteresting manner, i.e., the fold-change in expression is the same for most genes and is directly driven by the change in dosage. In the DE analysis, this systematic fold-change in expression is treated as a technical bias between samples and is removed by scaling normalization (e.g., TMM). The genes that are detected as DE are those that are (i) unaffected or less affected by changes in gene dosage, e.g., due to negative feedback; or (ii) affected more than one might expect from a simple change in dosage, e.g., due to positive feedback somewhere. The regulatory mechanisms driving these genes might be interesting - at least, moreso than a conclusion that most genes change in expression when you change the number of genome copies.
very interesting, thanks for the reflection Aaron.
Is it possible to compare gene expression by different ploidy genomes within the same species