Importing more than 100,000 at once into BiocFileCache
0
0
Entering edit mode
@pierrickroger-21304
Last seen 7 days ago
Italy

I've switched recently from custom cache system to BiocFileCache inside my biodb package. However, now, in one of the comming extension packages of biodb (biodbHmdb) I need to move more than 100,000 files at once into BiocFileCache. For this I call bfcadd() on each file inside a loop, and it takes forever to complete. Is there a faster way of achieving this or would it be possible to improve BiocFileCache for such a purpose?

BiocFileCache • 82 views
0
Entering edit mode

Do you really need to move all 100,000 files at the same time? It sounds like you're pre-loading the cache. To my mind a cache is useful for speeding up multiple similar operations, but not necessarily the first one. I don't know the context for biodbHmdb, but I'm wondering if it might be more effective to cache the result of a query after it has been executed.

This might also get more traction on the developers mailing list (https://stat.ethz.ch/mailman/listinfo/bioc-devel)