Importing more than 100,000 at once into BiocFileCache
0
0
Entering edit mode
@pierrickroger-21304
Last seen 11 months ago
Italy

I've switched recently from custom cache system to BiocFileCache inside my biodb package. However, now, in one of the comming extension packages of biodb (biodbHmdb) I need to move more than 100,000 files at once into BiocFileCache. For this I call bfcadd() on each file inside a loop, and it takes forever to complete. Is there a faster way of achieving this or would it be possible to improve BiocFileCache for such a purpose?

BiocFileCache • 806 views
ADD COMMENT
0
Entering edit mode

Do you really need to move all 100,000 files at the same time? It sounds like you're pre-loading the cache. To my mind a cache is useful for speeding up multiple similar operations, but not necessarily the first one. I don't know the context for biodbHmdb, but I'm wondering if it might be more effective to cache the result of a query after it has been executed.

This might also get more traction on the developers mailing list (https://stat.ethz.ch/mailman/listinfo/bioc-devel)

ADD REPLY

Login before adding your answer.

Traffic: 625 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6