Entering edit mode
Hi!
I'm using IdTaxa to classify eDNA sequences against various databases. For large databases, for example MIDORI or BOLD, training takes very long, often several days. Is there any way to speed this up, short of reducing the iterations (I use 3)? I removed the gap deletion and orienting parts already as they are not necessary. Maybe some parts in LearnTaxa could be parallelized, or has the function been optimized as much as possible already?
Cheers, Till
Hi Till, Which part is taking a long time: the part before the progress bar or during the progress bar? -- Erik
Hi Erik,
I think both? With all of BOLD I think the part before the progress bar takes about one day (removing groups also about that long), and the progress bar about 4 days, but I didn't actually keep track. I guess it's just a lot of sequences...
This is on an older Intel Xeon E5-2620.
Hi Till, It looks like you are training on millions of sequences, if I am interpreting your output correctly. I would expect this to take a long time: both training and classifying. There was a change that increased training speed a few versions ago, but if you are using the latest version of
DECIPHERthen that is as fast as it will go. SettingmaxChildren=0will provide some speedup to training, but may slow down classification depending on the training set. Your best bet is to useClusterize()to reduce redundant training sequences. There is negligible advantage to including very similar sequences with the same label when training. -- Erik