I don't have any problem running this on my end on a small subset.

Just to say again (for others reading this thread) your design matrix is really large (234 rows x 119 coefficients), so it makes sense this takes a while to fit with a GLM that has to iteratively reach the solution for each gene.

I tried running estimateDispersionGeneEst for the first 200 genes, and this takes 2.5 minutes. This is the longest step. The MAP step takes 1 minute, and Wald testing takes 1 minute. Extrapolating 4.5 minutes for 200 genes to 15,000 genes, and using 10 cores, this should take about 30 minutes (plus multicore overhead).

However, like I've said, for such a large design matrix I would instead use a linear model, such as limma-voom.

dds <- estimateSizeFactors(dds)
norm.cts <- counts(dds, normalized=TRUE)
rsnz <- rowSums(norm.cts >= 10)
table(rsnz >= 10) # here, 10 bc you have >200 samples in 4 groups
keep <- rsnz >= 10 

dds <- dds[keep,]
dds <- estimateSizeFactors(dds)

mm1 <- model.matrix(~ sex + sex:nested + sex:condition, colData(dds))
all.zero <- apply(mm1, 2, function(x) all(x == 0))
table(all.zero)
mm1 <- mm1[,-which(all.zero)]

system.time({
  dds2 <- estimateDispersionsGeneEst(dds[1:200,], modelMatrix=mm1)
  })

system.time({
  dds2 <- estimateDispersionsFit(dds2)
  })

system.time({
  dds2 <- estimateDispersionsMAP(dds2, modelMatrix=mm1)
  })

system.time({
  dds2 <- nbinomWaldTest(dds2, modelMatrix=mm1)
  })


> sessionInfo()
 R version 3.3.2 (2016-10-31)
 Platform: x86_64-pc-linux-gnu (64-bit)
 Running under: Ubuntu 16.10

 locale:
  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8
  [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C
 [10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

 attached base packages:
 [1] parallel  stats4    stats     graphics  grDevices datasets  utils     methods   base

 other attached packages:
  [1] DESeq2_1.14.1              SummarizedExperiment_1.4.0 Biobase_2.34.0
  [4] GenomicRanges_1.26.1       GenomeInfoDb_1.10.1        IRanges_2.8.1
  [7] S4Vectors_0.12.0           BiocGenerics_0.20.0        rmarkdown_1.2
 [10] magrittr_1.5               knitr_1.15.1               devtools_1.12.0
 [13] BiocInstaller_1.24.0

 loaded via a namespace (and not attached):
  [1] genefilter_1.56.0    locfit_1.5-9.1       splines_3.3.2        lattice_0.20-34
  [5] colorspace_1.3-1     htmltools_0.3.5      survival_2.40-1      XML_3.98-1.5
  [9] foreign_0.8-67       withr_1.0.2          DBI_0.5-1            BiocParallel_1.8.1
 [13] RColorBrewer_1.1-2   plyr_1.8.4           stringr_1.1.0        zlibbioc_1.20.0
 [17] munsell_0.4.3        gtable_0.2.0         memoise_1.0.0        evaluate_0.10
 [21] latticeExtra_0.6-28  geneplotter_1.52.0   AnnotationDbi_1.36.0 htmlTable_1.7
 [25] Rcpp_0.12.8          acepack_1.4.1        xtable_1.8-2         scales_0.4.1
 [29] backports_1.0.4      Hmisc_4.0-0          annotate_1.52.0      XVector_0.14.0
 [33] gridExtra_2.2.1      ggplot2_2.2.0        digest_0.6.10        stringi_1.1.2
 [37] grid_3.3.2           rprojroot_1.1        tools_3.3.2          bitops_1.0-6
 [41] lazyeval_0.2.0       RCurl_1.95-4.8       tibble_1.2           RSQLite_1.1
 [45] Formula_1.2-1        cluster_2.0.5        Matrix_1.2-7.1       data.table_1.9.8
 [49] assertthat_0.1       rpart_4.1-10         nnet_7.3-12          compiler_3.3.2
 >