Hi,

I'm trying to download the raw metadata files for recount3 via the IDIES links like the ones listed on http://rna.recount.bio/docs/raw-files.html (e.g. http://idies.jhu.edu/recount3/data/human/data_sources/sra/metadata/65/SRP107565/sra.recount_pred.SRP107565.MD.gz) but am getting a "Page can't be found" error:

I was able to download the data just last week, but have been unable to for the past 5-6 days. To be transparent, I'm currently running a Python script that is attempting to go through the IDIES links one by one and download all the metadata files. A few weeks ago, I did something similar when I ran a script to download all the count files, and the same thing happened where the count file download links broke for a couple days, but then came back. I'm guessing something similar is happening with the metadata files. Perhaps my mass download is triggering something?

Thanks for the help!

Hi,

Thank you for your interest in recount3 (and recount2).

We've been documenting this issue at https://github.com/LieberInstitute/recount3/issues/29. The IDIES link you were using changed recently to https://sciserver.org/public-data/recount3/data. However, we now have a new (2nd) host thanks to AWS at https://registry.opendata.aws/recount/. This is now the default host used by our load balancer duffel (https://github.com/nellore/digitalocean-duffel). recount3 version 1.9.1 documents these new hosts https://github.com/LieberInstitute/recount3/commit/6cf18f316123695b6a93c2049ab499b00d6c2acf.

Note that you need TLS version 1.2 or newer which most people have. If you encounter any new issues, please let us know at https://github.com/LieberInstitute/recount3/issues.

Please help us share this announcement

Very happy day for the recount project (https://t.co/U6IqgxYHaI) as @awscloud now hosts our data through their Open Data program. By default, we now direct downloads to S3 URLs starting with https and pointing into a recount-opendata bucket. 1/2

— @benlangmead@genomic.social (@BenLangmead) January 11, 2023

Thanks! Leo

Hi again,

If you are a Windows user, duffel now fully works on that operating system. That is, the duffel access issue has now been addressed by the internal switch from RCurl::url.exists() to httr::http_error(). You can gain access to these updates by installing recount3 version 1.10.2 (bioc-release aka 3.17) or 1.11.2 (bioc-devel aka 3.18).

duffel currently points to https://registry.opendata.aws/recount/ instead of IDIES.

Best, Leo