Hi all, I have millions like 100M DNA reads each of which is ~150nt, some of them are duplicate. Is there any way to group the same sequences into one and count the number, like unique() function in R, but with the occurrence of read and also more efficient? Also, if I want to cluster these 100M reads based on their similarity, like editor distance or some distance <=2, is there some function or package can be used? Thank you! Xiaohui

You can use the aggregate function, i.e. if X is a vector of your sequences, i.e. X[i] is a character containing your i-th sequence (i=1,2,...,100M) then do y <- aggregate(X,list(X),length) then y is a two columns data.frame with column 1 containing the sequence and column 2 containing the count. If this is too slow in R, use sort -u on Unix (like Wei suggested) to get the sequences sorted and unique and then write a simple C program which runs over all your sequences and adds 1 to count[i] where i is this sequence's index in the unique and sorted list (you can use binary search to get that i). --- On Tue, 16/11/10, Xiaohui Wu <wux3 at="" muohio.edu=""> wrote: > From: Xiaohui Wu <wux3 at="" muohio.edu=""> > Subject: [BioC] Group millions of the same DNA sequences? > To: "bioconductor at stat.math.ethz.ch" <bioconductor at="" stat.math.ethz.ch=""> > Received: Tuesday, 16 November, 2010, 9:46 PM > Hi all, > > I have millions like 100M DNA reads each of which is > ~150nt, some of them are duplicate. Is there any way to > group the same sequences into one and count the number, like > unique() function in R, but with the occurrence of read and > also more efficient? > Also, if I want to cluster these 100M? reads based on > their similarity, like editor distance or some distance > <=2, is there some function or package can be used? > Thank you! > > Xiaohui > > _______________________________________________ > Bioconductor mailing list > Bioconductor at stat.math.ethz.ch > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor >

On 11/16/2010 02:46 AM, Xiaohui Wu wrote: > Hi all, > > I have millions like 100M DNA reads each of which is ~150nt, some of them are duplicate. Is there any way to group the same sequences into one and count the number, like unique() function in R, but with the occurrence of read and also more efficient? dna = Biostrings::DNAStringSet(<reads>) ShortRead::tables(dna, Inf)[["top"]] for this; also selectMethod(tables, "DNAStringSet") to see the code if the format (named list) is not to your liking. Martin > Also, if I want to cluster these 100M reads based on their similarity, like editor distance or some distance <=2, is there some function or package can be used? > Thank you! > > Xiaohui > > _______________________________________________ > Bioconductor mailing list > Bioconductor at stat.math.ethz.ch > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor -- Computational Biology Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N. PO Box 19024 Seattle, WA 98109 Location: M1-B861 Telephone: 206 667-2793

Dear Xiaohui: The unix command sort (which is also available on Mac) can group your sequences. But I guess you will have to write some code to count the numbers for each distinct sequence. I have written some C code to do the similar thing and I will be happy to share them. But I am not sure if your data format is the same as mine. I am not aware of any R functions which can cluster strings. I image it will be extremely slow if there are such functions. Hope this helps. Cheers, Wei On Nov 16, 2010, at 9:46 PM, Xiaohui Wu wrote: > Hi all, > > I have millions like 100M DNA reads each of which is ~150nt, some of them are duplicate. Is there any way to group the same sequences into one and count the number, like unique() function in R, but with the occurrence of read and also more efficient? > Also, if I want to cluster these 100M reads based on their similarity, like editor distance or some distance <=2, is there some function or package can be used? > Thank you! > > Xiaohui > > _______________________________________________ > Bioconductor mailing list > Bioconductor at stat.math.ethz.ch > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor ______________________________________________________________________ The information in this email is confidential and intend...{{dropped:6}}

Hi Wei, Thank you for your reply! I'll be very appreciated if you could send me your C code for reference. Xiaohui ------------------------------------------------------------- ????Wei Shi ?????2010-11-17 05:55:04 ????Wu, Xiaohui Ms. ???bioconductor at stat.math.ethz.ch ???Re: [BioC] Group millions of the same DNA sequences? Dear Xiaohui: The unix command sort (which is also available on Mac) can group your sequences. But I guess you will have to write some code to count the numbers for each distinct sequence. I have written some C code to do the similar thing and I will be happy to share them. But I am not sure if your data format is the same as mine. I am not aware of any R functions which can cluster strings. I image it will be extremely slow if there are such functions. Hope this helps. Cheers, Wei On Nov 16, 2010, at 9:46 PM, Xiaohui Wu wrote: > Hi all, > > I have millions like 100M DNA reads each of which is ~150nt, some of them are duplicate. Is there any way to group the same sequences into one and count the number, like unique() function in R, but with the occurrence of read and also more efficient? > Also, if I want to cluster these 100M reads based on their similarity, like editor distance or some distance <=2, is there some function or package can be used? > Thank you! > > Xiaohui > > _______________________________________________ > Bioconductor mailing list > Bioconductor at stat.math.ethz.ch > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor ______________________________________________________________________ The information in this email is confidential and intended solely for the addressee. You must not disclose, forward, print or use it without the permission of the sender. ______________________________________________________________________ .

Thank you Aaron! Till now, sort and uniq may be the easiest way to do this. For clustering, I don't think assembler is suitable for my case. I want to cluster similar reads to get different clusters, each cluster has some reads, and do further analysis. Xiaohui ------------------------------------------------------------- ????Aaron Mackey ?????2010-11-17 22:58:57 ????Wu, Xiaohui Ms. ???bioconductor at stat.math.ethz.ch ???Re: [BioC] Group millions of the same DNA sequences? sort -u | uniq -c will do the counting for you. but as these are sequence data, you should probably be concerned about indels (gaps) as well as mismatches, so the "edit distance" is not a great clustering metric. You could, of course, calculate all-vs-all Needleman-Wunsch pairwise alignments, from which you could count edit operations (counting any indel as one operation, regardless of length), and cluster by some edit operation threshold, but that would be computationally expensive. what is it you're trying to accomplish with these reads? Assembly? If so, just give your reads to an assembler (MIRA, Velvet, etc.) and let the assembler do this clustering for you. -Aaron 2010/11/16 Xiaohui Wu <wux3 at="" muohio.edu<mailto:wux3="" at="" muohio.edu="">> Hi Wei, Thank you for your reply! I'll be very appreciated if you could send me your C code for reference. Xiaohui ------------------------------------------------------------- ????Wei Shi ?????2010-11-17 05:55:04 ????Wu, Xiaohui Ms. ???bioconductor at stat.math.ethz.ch<mailto:bioconductor at="" stat.math.ethz.ch=""> ???Re: [BioC] Group millions of the same DNA sequences? Dear Xiaohui: The unix command sort (which is also available on Mac) can group your sequences. But I guess you will have to write some code to count the numbers for each distinct sequence. I have written some C code to do the similar thing and I will be happy to share them. But I am not sure if your data format is the same as mine. I am not aware of any R functions which can cluster strings. I image it will be extremely slow if there are such functions. Hope this helps. Cheers, Wei On Nov 16, 2010, at 9:46 PM, Xiaohui Wu wrote: > Hi all, > > I have millions like 100M DNA reads each of which is ~150nt, some of them are duplicate. Is there any way to group the same sequences into one and count the number, like unique() function in R, but with the occurrence of read and also more efficient? > Also, if I want to cluster these 100M reads based on their similarity, like editor distance or some distance <=2, is there some function or package can be used? > Thank you! > > Xiaohui > > _______________________________________________ > Bioconductor mailing list > Bioconductor at stat.math.ethz.ch<mailto:bioconductor at="" stat.math.ethz.ch=""> > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor ______________________________________________________________________ The information in this email is confidential and intended solely for the addressee. You must not disclose, forward, print or use it without the permission of the sender. ______________________________________________________________________ . _______________________________________________ Bioconductor mailing list Bioconductor at stat.math.ethz.ch<mailto:bioconductor at="" stat.math.ethz.ch=""> https://stat.ethz.ch/mailman/listinfo/bioconductor Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor