Hi,
I have more of a suggestion rather than a question. I used the unmix() function from the DEseq2 package on my data and kept getting the same error
Error in optim(par = rep(1, ncol(pure)), fn = sumLossVST, gr = NULL, :
non-finite finite-difference value
A search for the error led me to a PDF where a solution is described, see page 7. I retrieved the unmix() code from GitHub and changed line 98 to
method="L-BFGS-B", lower=c(.0001,.0001),upper=c(Inf,Inf))$par
The function now worked for me. Of course now all values that would be zero are 0.001 or slightly higher but otherwise the output I got makes sense. Just wanted to report this because I'm not a statistician and don't know if this could have changed my output in any significant way. If not this might be helpful to people encountering the same problem.