Hi,
This question may not be related to bioconductor, but I get an error running TOMplot in the WGCNA package. I got the heatmap and the color bars, but the dendrograms on the sides did not finish:
> TOMplot(plotTOM, geneTree, moduleColors, main = "Network heatmap plot, all genes")
Error in structure(NextMethod("[["), class = "dendrogram") :
node stack overflow
Error during wrapup: node stack overflow
Error in dev.flush() : node stack overflow
Error during wrapup: node stack overflow
> sessionInfo()
R version 3.1.0 (2014-04-10)
Platform: x86_64-apple-darwin10.8.0 (64-bit)
locale:
[1] C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] WGCNA_1.41-1 flashClust_1.01-2 dynamicTreeCut_1.62 impute_1.38.1
loaded via a namespace (and not attached):
[1] Formula_1.1-2 Hmisc_3.14-4 R.methodsS3_1.6.1 RColorBrewer_1.0-5
[5] Rcpp_0.11.2 cluster_1.15.3 codetools_0.2-9 doParallel_1.0.8
[9] foreach_1.4.2 grid_3.1.0 iterators_1.0.7 lattice_0.20-29
[13] latticeExtra_0.6-26 matrixStats_0.10.0 parallel_3.1.0 plyr_1.8.1
[17] reshape_0.8.5 splines_3.1.0 survival_2.37-7 tools_3.1.0

I've never used the WGCNA, but I'm curious if you can try reducing the size of your dataset and replotting it to see if you get the same errors?
I think you're right that it has to do with the size of the dataset. I have about 4400 genes. Will try to reduce it. Thanks.