You cannot use glmQLFit() if you don't have replicates.
In the absence of replicates, the only choices are exactTest() or glmFit() followed by glmLRT().
In the absence of replicates, you have to set the dispersion yourself. If you don't have any DE genes and you think you should, then apparently you have set the dispersion too high. This has nothing to do with trying to change the edgeR functions you are using.
glmQLFit/glmQLFTest can be used almost anywhere that glmFit/glmLRT can, but it's not going to give you a longer list of significant genes than glmFit/glmLRT. In fact, it's almost certain to do the opposite.
Performing a differential expression test with no replicates is a last resort method, and you should not be surprised when it gives you no significant results.