I guess you mean extract the names of genes that correspond to parts of the dendrogram?
It depends on which method you use to create the dendrogram. If you've got one, such as pheatmap, that returns the hclust objects, then you can use cutree to get the labels of the genes. So it's just a question of extracting the hclust object from your method of generating the clusters, or recreating the same dendrogram if your clustering plot doesn't return it. e.g
pl <- pheatmap(...) hc <- pl$tree_row lbl <- cutree(hc, 5) # you'll need to change '5' to the number of gene-groups you're interested in which(lbl==1) # find genes corresponding to first group, ...
or in ComplexHeatmap
pl <- Heatmap(...) hc <- as.hclust(row_dend(pl)[])
If you can't find a method for extracting the dendrogram, then you'll need to look at exactly how the clustering is generating, and simulate all steps in its construction (ie is the matrix scaled, what distance metric is it using, what agglomeration method, ...).