I am interested to selecte the top genes that carry signal in the dataset. What is the difference between the most variable genes across samples from these two functions?

library("genefilter")
topVarGenes <- head(order(-rowVars(assay(vsd))),30)

#Different from the top 30 from here (some are, some aren't): 
DEG <- subset(res, padj <0.1)

• Is this because res have the results from the DESeq2 without the VST transformation?
  • Would it make any sense to obtain the DEG after vst to obtain the most variable genes after normalisation? (this is what topVarGenes does)
  • If they are different, which one are to use for what?

That is actually pretty simple. rowVars selects genes based on the row-wise variance, and the DEG results are based on the entire differential expression framework which is described both in the paper and the vignette. The latter would be more reliable in terms of which genes have actual statistical support to be DE, and the first one is more of a quick/dirty feature selection, e.g. for exploratory purposes such as PCA. The DEG have nothing to do with the vst function whereas it makes sense to run rowVars on the output of vst as this corrects for the biased mean/variance trend and puts data on the log scale. If you want DEGs then follow the standard pipeline in the vignette (which explicitely mentions that vst is not used throughout the DE pipeline), and if you want a quick feature selection for QC / exploration then use rowVars.

Edit: Oh yeah, asked before: https://www.biostars.org/p/472945/

What is unclear, what don't you understand, there are in fact multiple threads where you asked/commented on that over at biostars. Don't overthink it. I always do it like this: Use the top variable genes for a quick initial QC, and for anything else use the DEGs (heatmaps, classifications...). Does that make sense to you?