Question

Download errors during Bioconductor 3.12 update

1

Entering edit mode

Michael Steinbaugh ▴ 90

@mjsteinbaugh

Last seen 12 months ago

Cambridge, MA

I'm seeing download errors with Bioconductor 3.12 for some packages currently, primarily some large data packages.

Here's an example error with reactome.db:

install("reactome.db")

Bioconductor version 3.12 (BiocManager 1.30.10), R 4.0.3 (2020-10-10)
Installing package(s) 'reactome.db'
installing the source package ‘reactome.db’

trying URL 'https://bioconductor.org/packages/3.12/data/annotation/src/contrib/reactome.db_1.74.0.tar.gz'
Content type 'application/x-gzip' length 430092992 bytes (410.2 MB)
================
downloaded 137.5 MB

Error in download.file(url, destfile, method, mode = "wb", ...) :
  (converted from warning) downloaded length 144140511 != reported length 430092992
Calls: install ... tryCatchList -> tryCatchOne -> doTryCatch -> download.file

I'm also seeing this with some of the BSgenome packages.

sessionInfo()

R version 4.0.3 (2020-10-10)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.7

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRblas.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] bb8_0.2.32

loaded via a namespace (and not attached):
[1] compiler_4.0.3      BiocManager_1.30.10 magrittr_1.5
[4] tools_4.0.3         crayon_1.3.4        remotes_2.2.0
[7] rlang_0.4.8

reactome.db Bioconductor BiocManager • 2.9k views

ADD COMMENT • link updated 4.1 years ago by Martin Morgan 25k • written 4.1 years ago by Michael Steinbaugh ▴ 90

1

Entering edit mode

This 'works for me'. Usually the error download length XXX != reported length YYY indicates that an internet connectivity issue, and the best first line of attack is to try again, perhaps from a better connection. A second approach might be to visit the URL in, e.g., a browser https://bioconductor.org/packages/3.12/data/annotation/src/contrib/reactome.db_1.74.0.tar.gz to download the file, and then to install it locally, install.packages("reactome.db_1.74.0.tar.gz", repos = NULL, type = "source").

ADD REPLY • link 4.1 years ago Martin Morgan 25k

0

Entering edit mode

Thanks Martin, I'm still seeing it this morning on my home connection. I'm guessing it's an ISP / CDN issue, but just wanted to let you guys know. I'll see if I can replicate it on an Amazon EC2 instance.

ADD REPLY • link 4.1 years ago Michael Steinbaugh ▴ 90

1

Entering edit mode

Yeah this seems to be an issue specifically with the internal download.packages call. I can download the package manually using curl or wget fine and install directly from the tarball. The download only fails inside of the R install command.

ADD REPLY • link 4.1 years ago Michael Steinbaugh ▴ 90

2

Entering edit mode

Good catch. The method used to download packages is influenced by the global option download.file.method as documented on ?download.file. So it would be interesting to know whether

## "curl" or "wget" or "libcurl" or "wininet" (Windows only)
options(download.file.method = "curl")
BiocManager::install("reactome.db")

succeeds.

ADD REPLY • link 4.1 years ago Martin Morgan 25k

0

Entering edit mode

Thanks this is helpful for debugging.

libcurl seems to be the culprit on my machine.

options(download.file.method = "libcurl")
BiocManager::install("geneLenDataBase")

> BiocManager::install("geneLenDataBase")
Bioconductor version 3.12 (BiocManager 1.30.10), R 4.0.3 (2020-10-10)
Installing package(s) 'geneLenDataBase'
installing the source package ‘geneLenDataBase’

trying URL 'https://bioconductor.org/packages/3.12/data/experiment/src/contrib/geneLenDataBase_1.26.0.tar.gz'
Content type 'application/x-gzip' length 100436635 bytes (95.8 MB)
================================================
downloaded 92.8 MB

Warning in download.file(url, destfile, method, mode = "wb", ...) :
  downloaded length 97321263 != reported length 100436635
Calls: <Anonymous> ... tryCatchList -> tryCatchOne -> doTryCatch -> download.file
Warning in download.file(url, destfile, method, mode = "wb", ...) :
  URL 'https://bioconductor.org/packages/3.12/data/experiment/src/contrib/geneLenDataBase_1.26.0.tar.gz': Timeout of 5 seconds was reached
Calls: <Anonymous> ... tryCatchList -> tryCatchOne -> doTryCatch -> download.file
Error in download.file(url, destfile, method, mode = "wb", ...) :
  download from 'https://bioconductor.org/packages/3.12/data/experiment/src/contrib/geneLenDataBase_1.26.0.tar.gz' failed
Warning in download.packages(pkgs, destdir = tmpd, available = available,  :
  download of package ‘geneLenDataBase’ failed
Calls: <Anonymous> ... doTryCatch -> install.packages -> download.packages

curl works, but with some issues.

options(download.file.method = "curl")
BiocManager::install("geneLenDataBase")

Installing package(s) 'geneLenDataBase'
Warning: unable to access index for repository https://bioconductor.org/packages/3.12/data/annotation/bin/macosx/contrib/4.0:
  Found continuation line starting '    <script type="te ...' at begin of record.
Warning: unable to access index for repository https://bioconductor.org/packages/3.12/data/experiment/bin/macosx/contrib/4.0:
  Found continuation line starting '    <script type="te ...' at begin of record.
Warning: unable to access index for repository https://bioconductor.org/packages/3.12/workflows/bin/macosx/contrib/4.0:
  Found continuation line starting '    <script type="te ...' at begin of record.
Warning: unable to access index for repository https://r.acidgenomics.com/bin/macosx/contrib/4.0:
  Line starting '<!DOCTYPE html> ...' is malformed!
installing the source package ‘geneLenDataBase’

wget also works but is a little overly verbose with the HTTP headers.

options(download.file.method = "wget")
BiocManager::install("geneLenDataBase")

ADD REPLY • link 4.1 years ago Michael Steinbaugh ▴ 90

0

Entering edit mode

A second work-around might be to change the option "timeout", e.g., options(timeout = 180L) as described on the help page of ?download.files. It would be good to know if this also works.

ADD REPLY • link 4.1 years ago Martin Morgan 25k