optimizing VariantTools\' performance
1
0
Entering edit mode
Guest User ★ 13k
@guest-user-4897
Last seen 9.6 years ago
I am running VariantTools on 40 not particularly large bam files (~700 Mb). When running 5 in parallel on the same machine (12 cores, 48 Gig RAM), each running on a single core, it took several hours for the first one to finish, and the other 4 are still running. In total, the remaing 4 are taking up 70% of the RAM. Does anyone have suggestions for how to run VariantTools for optimum speed? I can increase the number of cores, but then I have to run fewer files because otherwise I run out of memory. And if I run 5 at a time each with a single core, I'm coming close to using up all the memory, and I don't know if that will really slow things down and if it would therefore be better to run just 4 at a time. Any suggestions are appreciated. Thank you. Eric -- output of sessionInfo(): These are running in batch mode so I don't have that information. -- Sent via the guest posting facility at bioconductor.org.
VariantTools VariantTools • 809 views
ADD COMMENT
0
Entering edit mode
@michael-lawrence-3846
Last seen 2.3 years ago
United States
I think you should try increasing the cores (for the tally step). The memory consumption in that step is mostly due to the summaries. All the summaries need to be generated prior to the filtering, so you might as well calculate them more quickly. Another idea is to parallelize at a higher level, so that each core is responsible for both tallying and filtering (for, say, one chromosome). This would be more work on your part, though. Eventually VariantTools needs to make this easier. Michael On Thu, Jun 13, 2013 at 2:02 PM, Eric Foss [guest] <guest@bioconductor.org>wrote: > > I am running VariantTools on 40 not particularly large bam files (~700 > Mb). When running 5 in parallel on the same machine (12 cores, 48 Gig RAM), > each running on a single core, it took several hours for the first one to > finish, and the other 4 are still running. In total, the remaing 4 are > taking up 70% of the RAM. Does anyone have suggestions for how to run > VariantTools for optimum speed? I can increase the number of cores, but > then I have to run fewer files because otherwise I run out of memory. And > if I run 5 at a time each with a single core, I'm coming close to using up > all the memory, and I don't know if that will really slow things down and > if it would therefore be better to run just 4 at a time. Any suggestions > are appreciated. > > Thank you. > > Eric > > -- output of sessionInfo(): > > These are running in batch mode so I don't have that information. > > -- > Sent via the guest posting facility at bioconductor.org. > > _______________________________________________ > Bioconductor mailing list > Bioconductor@r-project.org > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: > http://news.gmane.org/gmane.science.biology.informatics.conductor > [[alternative HTML version deleted]]
ADD COMMENT

Login before adding your answer.

Traffic: 656 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6