calculateTSNE with large dataset
1
0
Entering edit mode
alaihd ▴ 10
@luyang1005-13391
Last seen 2.1 years ago
United States

Hi developer,

I have a dataset with more than 30,000,000 cells * 30 genes for multiple samples. I want to use runTSNE or calculateTSNE function to reduce dimension. May I please know a feasible way? Much appreciated!

runTSNE calculateTSNE scater • 1.7k views
ADD COMMENT
0
Entering edit mode

Do you truly mean 30 million cells? That is orders of magnitude larger than any scRNA-seq dataset I have seen. I've never seen a t-SNE plot for more than about 200,000 cells at a time.

ADD REPLY
0
Entering edit mode

Gordon Smyth Yes, it is about 30million cells, which comes from more than 100 samples. Not scRNA-seq, it comes from mass cytometry.

ADD REPLY
1
Entering edit mode

An alternative would be to cluster your cells first and then downsample your cells by stratifying per cluster.

ADD REPLY
1
Entering edit mode
@alanocallaghan-14291
Last seen 24 days ago
United Kingdom

You can use snifter, which has a wrapper in scater. See the example code below. I think UMAP also scales pretty well with the number of cells. Either is likely to take several hours.

Be aware that I have 32GB of RAM on my local machine and this runs out of memory very quickly. It's been running a few hours on the cluster, not sure how long it'll take in total.

library("scater")
library("SingleCellExperiment")
library("DelayedArray")

mat <- DelayedArray(matrix(rnorm(30*30000000), nrow = 30))
sce <- SingleCellExperiment(assays = list(logcounts=mat))
sce <- runTSNE(sce, use_fitsne=TRUE)
sce <- runUMAP(sce)
ADD COMMENT
0
Entering edit mode

alan.ocallaghan Sounds like a feasible plan for me. Thanks! For runTSNE and runUMAP, they supply the "num_threads" parameter, do you think if I offer more threads, will the computation be faster?

ADD REPLY
0
Entering edit mode

For UMAP, yes; for TSNE, you probably want to use the FI-tSNE algorithm for which you'll want to use num_jobs instead of num_threads (sorry for the name mismatch, I'll patch that soon)

ADD REPLY
0
Entering edit mode

alan.ocallaghan Thanks a lot.

ADD REPLY

Login before adding your answer.

Traffic: 712 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6