Unable to download hg19.generanges as part of DMRcatedata package
1
0
Entering edit mode
@richardallenfriedmanbrooklyn-24118
Last seen 7 weeks ago
United States

Unable to download hg19.generanges as part of DMRcatedata package

Dear List,

I have been unable to download the entire hg19.generanges for DMRcatedata on a mac at work. I can download it at home, but my mac at work has more memory, so I would like to use the work mac.

Here is my session (I has to eliminate part of the sessionInfo() to stay within character limit. Please raise the character limit):

> results.ranges <- extractRanges(DMRs, genome = "hg19")
snapshotDate(): 2020-04-27
see ?DMRcatedata and browseVignettes('') for documentation
downloading 1 resources
retrieving 1 resource
  |                                                                                                                               |   0%Error: failed to load resource
  name: EH3132
  title: hg19.generanges
  reason: 1 resources failed to download
In addition: Warning messages:
1: download failed
  web resource path: 'https://experimenthub.bioconductor.org/fetch/3148'
  local file path: '/Users/friedman/Library/Caches/ExperimentHub/17d25be40bc3_3148'
  reason: transfer closed with 900409 bytes remaining to read 
2: bfcadd() failed; resource removed
  rid: BFC15
  fpath: 'https://experimenthub.bioconductor.org/fetch/3148'
  reason: download failed 
3: download failed
  hub path: 'https://experimenthub.bioconductor.org/fetch/3148'
  cache resource: 'EH3132 : 3148'
  reason: bfcadd() failed; see warnings() 
> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Mojave 10.14

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRblas.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats4    parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] DMRcatedata_2.6.0                                  ExperimentHub_1.14.2                              
 [3] AnnotationHub_2.20.2                               BiocFileCache_1.12.1                              
 [5] dbplyr_1.4.4                                       IlluminaHumanMethylation450kanno.ilmn12.hg19_0.6.0
 [7] ggplot2_3.3.2                                      clusterProfiler_3.16.1                            
 [9] xlsx_0.6.3                                         ChAMP_2.18.2                                      
[11] DT_0.15                                            IlluminaHumanMethylationEPICmanifest_0.3.0        
[13] Illumina450ProbeVariants.db_1.24.0                 FEM_3.15.0                                        
[15] graph_1.66.0                                       org.Hs.eg.db_3.11.4                               
[17] impute_1.62.0                                      igraph_1.2.5                                      
[19] corrplot_0.84                                      marray_1.66.0                                     
[21] limma_3.44.3                                       Matrix_1.2-18                                     
[23] AnnotationDbi_1.50.3                               ChAMPdata_2.20.0                                  
[25] DMRcate_2.2.2                                      minfi_1.34.0                                      
[27] bumphunter_1.30.0                                  locfit_1.5-9.4                                    
[29] iterators_1.0.12                                   foreach_1.5.0                                     
[31] Biostrings_2.56.0                                  XVector_0.28.0                                    
[33] SummarizedExperiment_1.18.2                        DelayedArray_0.14.1                               
[35] matrixStats_0.56.0                                 Biobase_2.48.0                                    
[37] GenomicRanges_1.40.0                               GenomeInfoDb_1.24.2                               
[39] IRanges_2.22.2                                     S4Vectors_0.26.1                                  
[41] BiocGenerics_0.34.0                               

l
> 

Thanks and best wishes, Rich Richard Friedman, Columbia Univeristy

DMRacte DMRcateranges ExperimentHub hg19.generanges • 1.1k views
ADD COMMENT
0
Entering edit mode
@james-w-macdonald-5106
Last seen 7 hours ago
United States

Can you get it by hand?

> library(ExperimentHub)
> hub <- ExperimentHub()
  |======================================================================| 100%

snapshotDate(): 2020-04-27

> z <- hub[["EH3132"]]
downloading 1 resources
retrieving 1 resource
  |======================================================================| 100%

loading from cache
> z
GRanges object with 55773 ranges and 1 metadata column:
             seqnames      ranges strand |      symbol
                <Rle>   <IRanges>  <Rle> | <character>
     DDX11L1     chr1 11869-14412      + |     DDX11L1
  MIR1302-10     chr1 29554-31109      + |  MIR1302-10
      OR4G4P     chr1 52473-54936      + |      OR4G4P
     OR4G11P     chr1 62948-63887      + |     OR4G11P
       OR4F5     chr1 69091-70008      + |       OR4F5
         ...      ...         ...    ... .         ...
       MT-TY     chrM   5826-5891      - |       MT-TY
      MT-TS1     chrM   7446-7514      - |      MT-TS1
      MT-ND6     chrM 14149-14673      - |      MT-ND6
       MT-TE     chrM 14674-14742      - |       MT-TE
       MT-TP     chrM 15956-16023      - |       MT-TP
  -------
  seqinfo: 25 sequences from an unspecified genome; no seqlengths

ADD COMMENT
0
Entering edit mode

Jim,

Thanks. What you suggest worked on my mac at home, but this is what I got on my work mac

> hub <- ExperimentHub()
snapshotDate(): 2020-04-27
> z <- hub[["EH3132"]]
see ?DMRcatedata and browseVignettes('DMRcatedata') for documentation
  |                                                                                                                               |   0%Error: failed to load resource
  name: EH3132
  title: hg19.generanges
  reason: 1 resources failed to download
In addition: Warning messages:
1: download failed
  web resource path: 'https://experimenthub.bioconductor.org/fetch/3148'
  local file path: '/Users/friedman/Library/Caches/ExperimentHub/19bc1aab7941_3148'
  reason: transfer closed with 900409 bytes remaining to read 
2: bfcadd() failed; resource removed
  rid: BFC17
  fpath: 'https://experimenthub.bioconductor.org/fetch/3148'
  reason: download failed 
3: download failed
  hub path: 'https://experimenthub.bioconductor.org/fetch/3148'
  cache resource: 'EH3132 : 3148'
  reason: bfcadd() failed; see warnings() 
> 

So I have the same problem.

Thanks and best wishes, Rich

1
Entering edit mode

Maybe something's weird with your cache. You could try removeResources to just get rid of EH3132, or removeCache to just nuke the whole thing and start again.

ADD REPLY
0
Entering edit mode

Jim,

I tried these. Are these commands correct?

> removeResources(ExperimentHub,EH3132)
Error in h(simpleError(msg, call)) : 
  error in evaluating the argument 'ids' in selecting a method for function 'removeResources': object 'EH3132' not found
> removeResources(ExperimentHub,EH3132)
Error in h(simpleError(msg, call)) : 
  error in evaluating the argument 'ids' in selecting a method for function 'removeResources': object 'EH3132' not found
> removeCache(ExperimentHub,ask=TRUE)
Error in (function (classes, fdef, mtable)  : 
  unable to find an inherited method for function 'hubCache' for signature '"function"'
> 

Thanks an best wishes,

Rich

Richard Friedman

1
Entering edit mode

Is removeResources(ExperimentHub, "EH3132") (quotes around 'EH3123') what you are looking for?

ADD REPLY
0
Entering edit mode

Martin,

I wasn;t able to download the hg19.generanges (EH3132) database on one of my macs (please see above). Jim MacDonald suggested that I remove EH3132 and try again. I had neglected to paste in my trying it with quotes,

> removeResources(ExperimentHub,"EH3132")
Error in (function (classes, fdef, mtable)  : 
  unable to find an inherited method for function 'dbfile' for signature '"function"'

I also didn;t get a result. My main goal is to get hg19.generanges (EH3132) for use with DMRcate.

Thanks and best wishes,

Rich

Richard Friedman

1
Entering edit mode

Oops, again, I guess that should have been removeResources(ExperimentHub(),"EH3132") with parentheses after ExperimentHub -- I'm interpreting the error message, rather than actually knowing what's going on.

ADD REPLY
1
Entering edit mode

Could you also send the results of doing the following:

httr::cache_info(httr::HEAD("https://experimenthub.bioconductor.org/fetch/3148")) and httr::GET("https://experimenthub.bioconductor.org/fetch/3148")

ADD REPLY
0
Entering edit mode

Lori,

I ran the commands that you recommended but still got the same error message when I ran the extractRanges command.

> httr::cache_info(httr::HEAD("https://experimenthub.bioconductor.org/fetch/3148"))
<cache_info>  http://s3.amazonaws.com/experimenthub/DMRcatedata/hg19.generanges.Rda
  Cacheable:     TRUE
  Last-Modified: Tue, 24 Sep 2019 11:28:06 GMT
  Etag:          "534bc86954286545e221a6ecad386d35"
> httr::GET("https://experimenthub.bioconductor.org/fetch/3148")
Response [http://s3.amazonaws.com/experimenthub/DMRcatedata/hg19.generanges.Rda]
  Date: 2020-09-09 14:15
  Status: 200
  Content-Type: binary/octet-stream
  Size: 900 kB
<BINARY BODY="">

Thanks and best wishes,

Rich

0
Entering edit mode

Martin, Here is my session removing the file. When I tried what Lori advised in her post, subsequent to running the command you gave, I still got the same error message as before.

> library(ExperimentHub)
> removeResources(ExperimentHub(),"EH3132")
  |===============================================================================================================================| 100%

Thanks and best wishes,

Rich

0
Entering edit mode

The error message you get should be instructive, so I would recommend paying closer attention to what R is telling you. Any time you get a message like that, where it says it can't find an inherited method, it implies that you are passing in an argument that isn't expected. And it says the signature is 'function', and we used ExperimentHub() to generate the 'hub' object, so to me at least that's a really big clue?

Anyhow, you can always check the help page, which says

'removeResources(hub, ids)': Removes listed ids from the local cache.
          ids are "AH" ids. Returns an updated hub object. To work with
          updated hub object suggested syntax is to reassign (ie. 'hub
          = removeResources(hub, "AH1")'). If ids are missing will
          remove all previously downloaded local resources.

Where it clearly states that the first argument is a 'hub object', which again is a pretty good clue. So what I did is

> hub <- ExperimentHub()
  |======================================================================| 100%

snapshotDate(): 2020-04-27
> removeResources(hub, "EH3132")
Removing the following from cache:
      EH3132

Which is what I was getting at in the first place.

ADD REPLY
0
Entering edit mode

And for completeness, right after removing that resource I got it again:

> z <- hub[["EH3132"]]
see ?DMRcatedata and browseVignettes('DMRcatedata') for documentation
downloading 1 resources
retrieving 1 resource
  |======================================================================| 100%

loading from cache

In which case I had to re-download because it had been removed, as compared to what you get if it's already in your cache:

> z <- hub[["EH3132"]]
see ?DMRcatedata and browseVignettes('DMRcatedata') for documentation
loading from cache
> 
ADD REPLY
0
Entering edit mode

And if you try to re-add the resource and still have trouble, could you show the full output from each of these commands httr::cache_info(httr::HEAD("https://experimenthub.bioconductor.org/fetch/3148")) and httr::GET("https://experimenthub.bioconductor.org/fetch/3148")

ADD REPLY
0
Entering edit mode

Jim and Lori, Thank you for the advice. I hope that the following is sufficiently complete to show you what is going on. I had to cut off most of the package listing for reasons of space, (and not attached):" for reasons of space

Thanks and best wishes, Rich

>   hub<-ExperimentHub()
snapshotDate(): 2020-04-27
>   removeResources(hub,"EH3132")
The following are not locally downloaded:
      EH3132
>  z<-hub[["EH3132"]] 
  |                                                                                                                               |   0%Error: failed to load resource
  name: EH3132
  title: hg19.generanges
  reason: 1 resources failed to download
In addition: Warning messages:
1: download failed
  web resource path: 'https://experimenthub.bioconductor.org/fetch/3148'
  local file path: '/Users/friedman/Library/Caches/ExperimentHub/39f8f020bdc_3148'
  reason: transfer closed with 900409 bytes remaining to read 
2: bfcadd() failed; resource removed
  rid: BFC20
  fpath: 'https://experimenthub.bioconductor.org/fetch/3148'
  reason: download failed 
3: download failed
  hub path: 'https://experimenthub.bioconductor.org/fetch/3148'
  cache resource: 'EH3132 : 3148'
  reason: bfcadd() failed; see warnings() 
> warnings()
Warning messages:
1: download failed
  web resource path: 'https://experimenthub.bioconductor.org/fetch/3148'
  local file path: '/Users/friedman/Library/Caches/ExperimentHub/39f8f020bdc_3148'
  reason: transfer closed with 900409 bytes remaining to read
2: bfcadd() failed; resource removed
  rid: BFC20
  fpath: 'https://experimenthub.bioconductor.org/fetch/3148'
  reason: download failed
3: download failed
  hub path: 'https://experimenthub.bioconductor.org/fetch/3148'
  cache resource: 'EH3132 : 3148'
  reason: bfcadd() failed; see warnings()
>   httr::cache_info(httr::HEAD("https://experimenthub.bioconductor.org/fetch/3148"))
<cache_info>  http://s3.amazonaws.com/experimenthub/DMRcatedata/hg19.generanges.Rda
  Cacheable:     TRUE
  Last-Modified: Tue, 24 Sep 2019 11:28:06 GMT
  Etag:          "534bc86954286545e221a6ecad386d35"
> httr::GET("https://experimenthub.bioconductor.org/fetch/3148")
Response [http://s3.amazonaws.com/experimenthub/DMRcatedata/hg19.generanges.Rda]
  Date: 2020-09-09 18:07
  Status: 200
  Content-Type: binary/octet-stream
  Size: 900 kB
<BINARY BODY="">
>   DMRs <- dmrcate(myAnno, lambda=1000, C=2)
Fitting chr1...
Fitting chr2...
Fitting chr3...
Fitting chr4...
Fitting chr5...
Fitting chr6...
Fitting chr7...
Fitting chr8...
Fitting chr9...
Fitting chr10...
Fitting chr11...
Fitting chr12...
Fitting chr13...
Fitting chr14...
Fitting chr15...
Fitting chr16...
Fitting chr17...
Fitting chr18...
Fitting chr19...
Fitting chr20...
Fitting chr21...
Fitting chr22...
Demarcating regions...
Done!
>  results.ranges <- extractRanges(DMRs, genome = "hg19")
snapshotDate(): 2020-04-27
see ?DMRcatedata and browseVignettes('DMRcatedata') for documentation
downloading 1 resources
retrieving 1 resource
  |                                                                                                                               |   0%Error: failed to load resource
  name: EH3132
  title: hg19.generanges
  reason: 1 resources failed to download
In addition: Warning messages:
1: download failed
  web resource path: 'https://experimenthub.bioconductor.org/fetch/3148'
  local file path: '/Users/friedman/Library/Caches/ExperimentHub/39f8724d592f_3148'
  reason: transfer closed with 900409 bytes remaining to read 
2: bfcadd() failed; resource removed
  rid: BFC21
  fpath: 'https://experimenthub.bioconductor.org/fetch/3148'
  reason: download failed 
3: download failed
  hub path: 'https://experimenthub.bioconductor.org/fetch/3148'
  cache resource: 'EH3132 : 3148'
  reason: bfcadd() failed; see warnings() 
> 
> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Mojave 10.14

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRblas.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats4    parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] DMRcatedata_2.6.0                                  IlluminaHumanMethylation450kanno.ilmn12.hg19_0.6.0
 [3] ggplot2_3.3.2                                      clusterProfiler_3.16.1                            
 [5] xlsx_0.6.3                                         ChAMP_2.18.2                                      


loaded via a namespace (and not attached):

0
Entering edit mode

This seems really odd. Can you download any other non DMRcatedata resources? hub[["EH166"]]

ADD REPLY
0
Entering edit mode

Lori,

The same thing happened with EH166.

Thanks and best wishes, Rich

> hub<-ExperimentHub()
snapshotDate(): 2020-04-27
> z<-hub[["EH166"]] 
see ?alpineData and browseVignettes('alpineData') for documentation
downloading 1 resources
retrieving 1 resource
  |                                                                                                                               |   0%Error: failed to load resource
  name: EH166
  title: ERR188297
  reason: 1 resources failed to download
In addition: Warning messages:
1: download failed
  web resource path: 'https://experimenthub.bioconductor.org/fetch/166'
  local file path: '/Users/friedman/Library/Caches/ExperimentHub/39f81a7102e9_166'
  reason: transfer closed with 167198 bytes remaining to read 
2: bfcadd() failed; resource removed
  rid: BFC23
  fpath: 'https://experimenthub.bioconductor.org/fetch/166'
  reason: download failed 
3: download failed
  hub path: 'https://experimenthub.bioconductor.org/fetch/166'
  cache resource: 'EH166 : 166'
  reason: bfcadd() failed; see warnings() 
> 
0
Entering edit mode

Could your institution be blocking downloads from AWS?

ADD REPLY
0
Entering edit mode

Lori,

I have a query into our IT department as to whether they are blocking the download. I am not experiencing this difficulty on my mac at home. I will repeat the query to IT mentioning that the source of the file is AWS.

Thanks and best wishes,

Rich

ADD REPLY
0
Entering edit mode

Lori,

I can download the file from the website via a browser but not via R as you have seen. Is there a kind of block that can block R downloads, but not Netscape downloads?

Thanks and best wishes, Rich

ADD REPLY
1
Entering edit mode

Its very baffling and I have not seen this before. Lets try a little more in depth debugging: Please do the following:

library(ExperimentHub)
hub = ExperimentHub()
debugonce(BiocFileCache:::.httr_download)
hub[["EH166"]]

Using debugonce will put you inside the function that tries to download the file so you can step line by line through the code. This will show the code and you press the return key to step through.
Once you see the following

Browse[2]> 
debug: response <- suppressWarnings({
    GET(websource, progress(con = stderr()), config = config, 
        write_disk(localfile, overwrite = TRUE), proxy, ...)
})

BEFORE pressing <enter> please investigate -- This should be the section of code that produces an ERROR So BEFORE pressing enter check the value of

websource
localfile
config
proxy

Please record these values and then press enter. Please include the output of those values and any ERROR message.

Let me know if you have any troubles trying to use debugonce.

We are thinking right now you may either not have write access or might be out of space in the caching location that is being used.

ADD REPLY
0
Entering edit mode

Lori,

Thank you for your advice. I apologize for the late response. I had several deadlines. I followed your instructions. Here are session excerpts:

Browse[2]> 
debug: response <- suppressWarnings({
    GET(websource, progress(con = stderr()), config = config, 
        write_disk(localfile, overwrite = TRUE), proxy, ...)
})
Browse[2]> websource
[1] "https://experimenthub.bioconductor.org/fetch/166"
Browse[2]> localfile
[1] "/Users/friedman/Library/Caches/ExperimentHub/fc9759b21af9_166"
Browse[2]> config
list()
Browse[2]> proxy
NULL

```

After some more stepping through

Error: failed to load resource
  name: EH166
  title: ERR188297
  reason: 1 resources failed to download
In addition: Warning messages:
1: download failed
  web resource path: ‘https://experimenthub.bioconductor.org/fetch/166’
  local file path: ‘/Users/friedman/Library/Caches/ExperimentHub/fc9759b21af9_166’
  reason: transfer closed with 164334 bytes remaining to read 
2: bfcadd() failed; resource removed
  rid: BFC25
  fpath: ‘https://experimenthub.bioconductor.org/fetch/166’
  reason: download failed 
3: download failed
  hub path: ‘https://experimenthub.bioconductor.org/fetch/166’
  cache resource: ‘EH166 : 166’
  reason: bfcadd() failed; see warnings() 
>

Does this tell you anything?

Best wishes,

Rich

ADD REPLY
1
Entering edit mode

I apologize for the issues and appreciate you debugging with me so we can figure it out. I'll give a few steps:
Does httr::GET("https://experimenthub.bioconductor.org/fetch/166") work?
If not directly after the call please do traceback() .
If successful, then try httr::GET("https://experimenthub.bioconductor.org/fetch/166", write_disk(tempfile())) Similarly if failed, traceback()
If successful, the try httr::GET("https://experimenthub.bioconductor.org/fetch/166", write_disk(paste0("/Users/friedman/Library/Caches/ExperimentHub/", basename(tempfile())))) If fail, traceback() Thanks

ADD REPLY
0
Entering edit mode

Lori,

Here is what I got,

The write_disk function is not present, although httr is installed.


> httr::GET("https://experimenthub.bioconductor.org/fetch/166")
Error in curl::curl_fetch_memory(url, handle = handle) : 
  transfer closed with 164334 bytes remaining to read
> traceback()
5: curl::curl_fetch_memory(url, handle = handle)
4: request_fetch.write_memory(req$output, req$url, handle)
3: request_fetch(req$output, req$url, handle)
2: request_perform(req, hu$handle$handle)
1: httr::GET("https://experimenthub.bioconductor.org/fetch/166")
> httr::GET("https://experimenthub.bioconductor.org/fetch/166", write_disk(tempfile()))
Error in write_disk(tempfile()) : could not find function "write_disk"
> traceback()
3: as.request(config)
2: request_build("GET", hu$url, as.request(config), ...)
1: httr::GET("https://experimenthub.bioconductor.org/fetch/166", 
       write_disk(tempfile()))
> 
> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Mojave 10.14

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRblas.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] httr_1.4.2     compiler_4.0.2 R6_2.4.1       tools_4.0.2    curl_4.3      
> 

> write_disk
Error: object 'write_disk' not found

Thanks and best wishes,

Rich

ADD REPLY
1
Entering edit mode

Its troubling that you are blocked from even doing httr::GET("https://experimenthub.bioconductor.org/fetch/166")

And you are sure your institution computer isn't behind a proxy that should be set at the start?

ADD REPLY
1
Entering edit mode

Is this an 'https'-related issue, e.g., does

httr::GET("http://experimenthub.bioconductor.org/fetch/166")
ADD REPLY
0
Entering edit mode

Martin,

Thanks for the suggestion. Here is what I get

> httr::GET("http://experimenthub.bioconductor.org/fetch/166")
Error in curl::curl_fetch_memory(url, handle = handle) : 
  transfer closed with 164334 bytes remaining to read
> traceback()
5: curl::curl_fetch_memory(url, handle = handle)
4: request_fetch.write_memory(req$output, req$url, handle)
3: request_fetch(req$output, req$url, handle)
2: request_perform(req, hu$handle$handle)
1: httr::GET("http://experimenthub.bioconductor.org/fetch/166")
> 

Does this session tell you anything? Best wishes, Rich

ADD REPLY
0
Entering edit mode

Next up, the http:// and https:// versions of

response = httr::GET("http://experimenthub.bioconductor.org/fetch/166", httr::verbose())
response = httr::GET("https://experimenthub.bioconductor.org/fetch/166", httr::verbose())
ADD REPLY
0
Entering edit mode

Martin, Here goes:


> response = httr::GET("http://experimenthub.bioconductor.org/fetch/166", httr::verbose())
-> GET /fetch/166 HTTP/1.1
-> Host: experimenthub.bioconductor.org
-> User-Agent: libcurl/7.54.0 r-curl/4.3 httr/1.4.2
-> Accept-Encoding: deflate, gzip
-> Accept: application/json, text/xml, application/xml, */*
-> 
<- HTTP/1.1 301 Moved Permanently
<- Date: Tue, 29 Sep 2020 18:20:07 GMT
<- Server: Apache/2.4.18 (Ubuntu)
<- Location: https://experimenthub.bioconductor.org/fetch/166
<- Content-Length: 352
<- Content-Type: text/html; charset=iso-8859-1
<- 
-> GET /fetch/166 HTTP/1.1
-> Host: experimenthub.bioconductor.org
-> User-Agent: libcurl/7.54.0 r-curl/4.3 httr/1.4.2
-> Accept-Encoding: deflate, gzip
-> Accept: application/json, text/xml, application/xml, */*
-> 
<- HTTP/1.1 302 Found
<- Date: Tue, 29 Sep 2020 18:20:07 GMT
<- Server: Apache/2.4.18 (Ubuntu)
<- X-XSS-Protection: 1; mode=block
<- X-Content-Type-Options: nosniff
<- X-Frame-Options: SAMEORIGIN
<- X-Powered-By: Phusion Passenger 4.0.46
<- Location: http://s3.amazonaws.com/experimenthub/alpineData/ERR188297.rda
<- Content-Length: 0
<- Status: 302 Found
<- Content-Type: text/html;charset=utf-8
<- 
-> GET /experimenthub/alpineData/ERR188297.rda HTTP/1.1
-> Host: s3.amazonaws.com
-> User-Agent: libcurl/7.54.0 r-curl/4.3 httr/1.4.2
-> Accept-Encoding: deflate, gzip
-> Accept: application/json, text/xml, application/xml, */*
-> 
<- HTTP/1.1 200 OK
<- x-amz-id-2: ty9w77louJgBqG3fiYeIBVrmWlpoO/Dwz+Ny6fdM2yRpJTLUuY2f7bL0Jx8kCBbp0kBTzE25SUk=
<- x-amz-request-id: 9757B982644C0B72
<- Date: Tue, 29 Sep 2020 18:20:08 GMT
<- Last-Modified: Thu, 21 Jul 2016 17:41:57 GMT
<- ETag: "1863a460a49ad03c8a6ed3579d8e63b5"
<- Accept-Ranges: bytes
<- Content-Type: application/octet-stream
<- Content-Length: 167198
<- Server: AmazonS3
<- 
Error in curl::curl_fetch_memory(url, handle = handle) : 
  transfer closed with 167198 bytes remaining to read
> traceback()
5: curl::curl_fetch_memory(url, handle = handle)
4: request_fetch.write_memory(req$output, req$url, handle)
3: request_fetch(req$output, req$url, handle)
2: request_perform(req, hu$handle$handle)
1: httr::GET("http://experimenthub.bioconductor.org/fetch/166", 
       httr::verbose())
> response = httr::GET("https://experimenthub.bioconductor.org/fetch/166", httr::verbose())
-> GET /fetch/166 HTTP/1.1
-> Host: experimenthub.bioconductor.org
-> User-Agent: libcurl/7.54.0 r-curl/4.3 httr/1.4.2
-> Accept-Encoding: deflate, gzip
-> Accept: application/json, text/xml, application/xml, */*
-> 
<- HTTP/1.1 302 Found
<- Date: Tue, 29 Sep 2020 18:21:20 GMT
<- Server: Apache/2.4.18 (Ubuntu)
<- X-XSS-Protection: 1; mode=block
<- X-Content-Type-Options: nosniff
<- X-Frame-Options: SAMEORIGIN
<- X-Powered-By: Phusion Passenger 4.0.46
<- Location: http://s3.amazonaws.com/experimenthub/alpineData/ERR188297.rda
<- Content-Length: 0
<- Status: 302 Found
<- Content-Type: text/html;charset=utf-8
<- 
-> GET /experimenthub/alpineData/ERR188297.rda HTTP/1.1
-> Host: s3.amazonaws.com
-> User-Agent: libcurl/7.54.0 r-curl/4.3 httr/1.4.2
-> Accept-Encoding: deflate, gzip
-> Accept: application/json, text/xml, application/xml, */*
-> 
<- HTTP/1.1 200 OK
<- x-amz-id-2: k8t2+wF29htUwq6zVdDVk760h+34XJW6VmtO+BSZ9WUBIccC9CZT1CGzkNwEhszXgUauBpcSu54=
<- x-amz-request-id: 691F1BC9338136B4
<- Date: Tue, 29 Sep 2020 18:21:21 GMT
<- Last-Modified: Thu, 21 Jul 2016 17:41:57 GMT
<- ETag: "1863a460a49ad03c8a6ed3579d8e63b5"
<- Accept-Ranges: bytes
<- Content-Type: application/octet-stream
<- Content-Length: 167198
<- Server: AmazonS3
<- 
Error in curl::curl_fetch_memory(url, handle = handle) : 
  transfer closed with 167198 bytes remaining to read
> traceback()
5: curl::curl_fetch_memory(url, handle = handle)
4: request_fetch.write_memory(req$output, req$url, handle)
3: request_fetch(req$output, req$url, handle)
2: request_perform(req, hu$handle$handle)
1: httr::GET("https://experimenthub.bioconductor.org/fetch/166", 
       httr::verbose())
>

Thanks and best wishes, Rich

ADD REPLY
0
Entering edit mode

Just popping out to this level so we have a bit more space.

It looks like the final file is http://s3.amazonaws.com/experimenthub/alpineData/ERR188297.rda. The command s something like

url <- "http://s3.amazonaws.com/experimenthub/alpineData/ERR188297.rda"
response = httr::GET(url, httr::verbose())

I get

> response
Response [http://s3.amazonaws.com/experimenthub/alpineData/ERR188297.rda]
  Date: 2020-10-19 15:22
  Status: 200
  Content-Type: application/octet-stream
  Size: 167 kB
<BINARY BODY="">
> raw <- httr::content(response)
> str(raw)
 raw [1:167198] 1f 8b 08 00 ...
ADD REPLY
0
Entering edit mode

Martin,

Apologies again. I looked at the support site everyday for a while, and the stopped so I missed your note. Here is my attempt

> library(ExperimentHub)
> url <- "http://s3.amazonaws.com/experimenthub/alpineData/ERR188297.rda"
> response = httr::GET(url, httr::verbose())
-> GET /experimenthub/alpineData/ERR188297.rda HTTP/1.1
-> Host: s3.amazonaws.com
-> User-Agent: libcurl/7.54.0 r-curl/4.3 httr/1.4.2
-> Accept-Encoding: deflate, gzip
-> Accept: application/json, text/xml, application/xml, */*
-> 
<- HTTP/1.1 200 OK
<- x-amz-id-2: UkGcU+scVtsmVheUc/jTSC3uWcGlNNvjCzjKmgsw+67RJc2GeFtcCKRywNwunaUeDepls26WFE0=
<- x-amz-request-id: 3A3710844D9D747D
<- Date: Wed, 23 Dec 2020 21:45:32 GMT
<- Last-Modified: Thu, 21 Jul 2016 17:41:57 GMT
<- ETag: "1863a460a49ad03c8a6ed3579d8e63b5"
<- Accept-Ranges: bytes
<- Content-Type: application/octet-stream
<- Content-Length: 167198
<- Server: AmazonS3
<- 
Error in curl::curl_fetch_memory(url, handle = handle) : 
  transfer closed with 167198 bytes remaining to read
>

Does this tell you anything? I suspect that despite Columbia IT's denials they are blocking AWS.

Thanks and best wishes, Rich

ADD REPLY
0
Entering edit mode

Martin,

I found out that curl doesn't work on mac at work (OS 10.14) but does work on my macs at home (OS 10.14.6 and 10.15). I also see that curl works from the cluster at work. So I think that I can solve the problem by updating the OS. Watch this space.

Thanks and best wishes,

Rich

ADD REPLY
0
Entering edit mode

Martin,

I got curl working on my work desktop, but I still get the same error message as before when Iuse experiment hub. I am going to update my mac os in the hope that doing so will help.

Thanks and Happy New Year, Rich

ADD REPLY

Login before adding your answer.

Traffic: 872 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6