To put more meat on my comment above, take a look at the code in the
vst function (look at
?vst and just type
vst at the console and hit return, the code will reveal itself)
You'll find that the
vst function does its magic on a subset of rows (ie. the size of the subset is equal to
nsub) then can apply it to the entire dataset.
If you call
varianceStabilizingTransform directly on your expression object
dds, it doesn't try to work on a subset of the genes in your expression matrix, and just does its thing on the whole thing.
Your problem is that your expression matrix doesn't have more than 1000 genes (rows) in it.