I have used MAST to obtain deviance residuals after accounting for the count detection rate in my cell samples and I found this to improve clustering and subtype identification. But do you have any opinion which distance measure is the best to use between different cells, when I cluster based on deviance residuals, rather than log2 transformed expression data?
I am asking because I do not get best results with cosine distance, which is the usual distance measure I choose. L1 norm, L2 norm, L3 norm all seem to work better, but I am not sure which one is the best and I could not think of a theoretical justification for either one yet, based on the fact that the underlying data are deviance residuals and not log2 expression data.
Thanks for your advice!