I'm using SingleR to annotate cell types in my single cell data but I recently discover a potential problem. It seems there is no minimum treshold for the correlations used for the transfer - meaning predictions will (almost) always be made even if the data does make sense.

To illustrate this point lets use at the example from the vignette

# BiocManager::install("SingleR")
library(SingleR)

### Load refrence data
hpca.se <- HumanPrimaryCellAtlasData()
hpca.se

# BiocManager::install("scRNAseq")
library(scRNAseq)
hESCs <- LaMannoBrainData('human-es')
hESCs <- hESCs[,1:100]

# BiocManager::install("scater")
library(scater)
hESCs <- logNormCounts(hESCs)

pred.hesc <- SingleR(test = hESCs, ref = hpca.se, labels = hpca.se$label.main)

sum(!is.na(pred.hesc$labels))
100
table(pred.hesc$labels)
Astrocyte Neuroepithelial_cell              Neurons 
       14                   81                    5

That works very nicely - we get 100 predictions from 5 cell types. (Although it can be debated whether these labels are good for embryonic stem cells).

Now lets scramble the data and see what happens.

### Scamble data
logCounts <- assay(hESCs, 'logcounts')
resampleData <- matrix(
    data = sample(unlist( logCounts )),
    ncol = ncol(          logCounts ),
    nrow = nrow(          logCounts )
)
rownames(resampleData) <- rownames( logCounts )
colnames(resampleData) <- colnames( logCounts )

### Verify no resemblance to original data
round( summary(sapply(1:ncol(logCounts), function(x) {
    cor(logCounts[,x],resampleData[,x])
})), 2)
 Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
-0.02    0.00    0.00    0.00    0.01    0.02

### Rerun SingleR
pred.rand <- SingleR(test = resampleData, ref = hpca.se, labels = hpca.se$label.main)
sum(!is.na(pred.rand$labels))
100
length(unique(pred.rand$labels))
29

So this time - with meaningless data we still get a hundred predictions (of very different cell types).

Is there a way to impose a minimum treshold on the correlations, possibly by comparing to scrambled data, to ensure cell types are only predicted when there is sufficient evidence?

Cheers Kristoffer

This is not a new problem. At least for default implementations of methods like k-NN or RFs or SVMs, no matter what you put in, you'll always get some assignments out. Sure, you could stick a minimum threshold onto the assignment score, but then how do you pick a threshold value in an intelligible manner? Scrambling will just generate a strawman, I've don't recall seeing real data actually hit a zero or negative correlation between different cell types; probably because many genes are constitutively expressed in all cell types, so you'll always get some level of positive correlation.

We have put some effort into identifying improbable assignments - see ?pruneScores for more details - but this makes a few assumptions of its own. We do not have a clean solution for the general problem of what happens when your cells do not exist in the reference. Well, that's not entirely true; the solution is to get a more comprehensive reference.

If you think you can do better, the correlations are reported in the output, so go for it.