Question: Error in getBM using biomaRt
0
gravatar for Oskar
11 weeks ago by
Oskar50
Oskar50 wrote:

Hi bioconductor support team,

I had run the same script for getting SNPs ID from a genome range by using biomaRt. Despite being able of getting the information needed for several ranges (about 26), there are two of them that didn't work.

Here is the script I've been using.

snp_mart = useMart(biomart = "ENSEMBL_MART_SNP", 
               dataset = "hsapiens_snp",
               host = "grch37.ensembl.org")

snp_id9 = getBM(attributes = c("refsnp_id", "allele","chr_name", "chrom_start", "chrom_end", "chrom_strand"), 
                            filters = c("chr_name", "start", "end"), 
                            values = list(9, 106356922, 107356689), 
                            mart = snp_mart)

snp_id8 = getBM(attributes = c("refsnp_id", "allele","chr_name", "chrom_start", "chrom_end", "chrom_strand"), 
                            filters = c("chr_name", "start", "end"), 
                            values = list(8, 143262170, 144261715), 
                            mart = snp_mart)

I am getting two Error messages when I re-run it - sometimes the first and sometimes the second. As I mentioned, I used exactly the same thing for 26 different ranges and everything went smooth.

Here the first one: Error in getBM(attributes = c("refsnpid", "allele", "chrname", "chrom_start", : The query to the BioMart webservice returned an invalid result: biomaRt expected a character string of length 1. Please report this on the support site at http://support.bioconductor.org

Here the second one: Error in curl::curlfetchmemory(url, handle = handle) : Timeout was reached: Operation timed out after 600000 milliseconds with 77645 bytes received

I would like to get some enlightenment to solve this issue. Thank you very much for your help in advance.

biomart • 154 views
ADD COMMENTlink modified 11 weeks ago by Mike Smith3.3k • written 11 weeks ago by Oskar50
Answer: Error in getBM using biomaRt
1
gravatar for Mike Smith
11 weeks ago by
Mike Smith3.3k
EMBL Heidelberg / de.NBI
Mike Smith3.3k wrote:

I'm not sure why you get differing error message, but I think the root cause is that these queries take too long to run and time out. When using the Ensembl web interface you get 5 minutes before a query dies, and you currently are allowed 10 minutes via biomaRt.

Query time is broadly related to the number of attributes you're returning and the number of values provided to the filters. In this case I think it's the size of the regions that is causing the issue. One approach would be to break the query down into smaller regions, submit each one, and then piece the results back together. Here's an example for your snp_id8:

## create a matrix where each row is a 100kb region
s1 <- seq(143262170, 144261715, by = 100000)
s2 <- c(s1[-1]-1, 144261715)
regions <- matrix(c(rep(chr, length(s1)), s1, s2), ncol = 3)

## wrapper function to be applied to each row of our matrix  
getBM_values <- function(values) {
  getBM(attributes = c("refsnp_id", "allele","chr_name", "chrom_start", "chrom_end", "chrom_strand"), 
        filters = c("chr_name", "start", "end"), 
        values = list(values[1], values[2], values[3]), 
        mart = snp_mart)
}

## query Ensembl for each region & combine results
res_list <- apply(regions, 1, getBM_values)
snp_id8 <- do.call(rbind, res_list)
> dim(snp_id8 )
[1] 218725      6
ADD COMMENTlink written 11 weeks ago by Mike Smith3.3k
1

Hi Mike - many thanks for your help. It works!!! I've tried creating blocks but not as matrix. Your strategy works just perfect. Than you very much indeed.

ADD REPLYlink written 11 weeks ago by Oskar50

Hi Mke - I think there is something wrong with both chromosome 8 and 9. I have retrieved the SNP IDs based on genome coordinates for chromosomes 1, 2, 3, 4, 5, 6, 7, 10, 11, 12, 13, 14, 18, 19, 20, 22, but I can't do it for chromosomes 8 and 9. I have also tried your script and mines in another terminal - considering the PC is the issue, but it wasn't successful as well.

I am getting this error:

> Error in getBM(attributes = c("refsnp_id", "allele", "chr_name", "chrom_start",  : 
  The query to the BioMart webservice returned an invalid result: biomaRt expected a character string of length 1. 
Please report this on the support site at http://support.bioconductor.org
Called from: getBM(attributes = c("refsnp_id", "allele", "chr_name", 
    "chrom_start", "chrom_end", "chrom_strand"), 
    filters = c("chr_name", "start", "end"), 
    values = list(values[1], values[2], values[3]), mart = snp_mart)
Browse[1]> Q
ADD REPLYlink written 8 weeks ago by Oskar50

Also, this debugging function pop up message:

function (attributes, filters = "", values = "", mart, curl = NULL, 
  checkFilters = TRUE, verbose = FALSE, uniqueRows = TRUE, 
  bmHeader = FALSE, quote = "\"") 
{
  martCheck(mart)
  if (missing(attributes)) 
    stop("Argument 'attributes' must be specified.")
  if (is.list(filters) && !missing(values)) 
    warning("Argument 'values' should not be used when argument 'filters' is a list and will be ignored.")
  if (is.list(filters) && is.null(names(filters))) 
    stop("Argument 'filters' must be a named list when sent as a list.")
  if (!is.list(filters) && filters != "" && missing(values)) 
    stop("Argument 'values' must be specified.")
  if (length(filters) > 0 && length(values) == 0) 
    stop("Values argument contains no data.")
  if (is.list(filters)) {
    values = filters
    filters = names(filters)
  }
  if (class(uniqueRows) != "logical") 
    stop("Argument 'uniqueRows' must be a logical value, so either TRUE or FALSE")
  callHeader <- TRUE
  xmlQuery = paste0("<Query virtualSchemaName="", 
    martVSchema(mart), "" uniqueRows="", as.numeric(uniqueRows), 
    "" count="0" datasetConfigVersion="0.6" header="", 
    as.numeric(callHeader), "" requestid="biomaRt"> <Dataset name="", 
    martDataset(mart), "">")
  invalid = !(attributes %in% listAttributes(mart, what = "name"))
  if (any(invalid)) 
    stop(paste("Invalid attribute(s):", paste(attributes[invalid], 
      collapse = ", "), "\nPlease use the function 'listAttributes' to get valid attribute names"))
  attributeXML = paste("<Attribute name="", attributes, 
    ""/>", collapse = "", sep = "")
  if (filters[1] != "" && checkFilters) {
    invalid = !(filters %in% listFilters(mart, what = "name"))
    if (any(invalid)) 
      stop(paste("Invalid filters(s):", paste(filters[invalid], 
        collapse = ", "), "\nPlease use the function 'listFilters' to get valid filter names"))
  }
  filterXmlList <- .generateFilterXML(filters, values, mart)
  resultList <- list()
  if (length(filterXmlList) > 1) {
    pb <- progress_bar$new(total = length(filterXmlList), 
      width = options()$width - 10, format = "Batch submitting query [:bar] :percent eta: :eta")
    pb$tick(0)
  }
  for (i in seq_along(filterXmlList)) {
    if (exists("pb")) {
      pb$tick()
    }
    filterXML <- filterXmlList[[i]]
    fullXmlQuery = paste(xmlQuery, attributeXML, filterXML, 
      "</Dataset></Query>", sep = "")
    if (verbose) {
      message(fullXmlQuery)
    }
    sep <- ifelse(grepl(x = martHost(mart), pattern = ".+\\?.+"), 
      "&", "?")
    postRes <- .submitQueryXML(host = paste0(martHost(mart), 
      sep), query = fullXmlQuery)
    if (verbose) {
      writeLines("#################\nResults from server:")
      print(postRes)
    }
    if (!(is.character(postRes) && (length(postRes) == 1L))) 
      stop("The query to the BioMart webservice returned an invalid result: biomaRt expected a character string of length 1. \nPlease report this on the support site at http://support.bioconductor.org")
    if (gsub("\n", "", postRes, fixed = TRUE, useBytes = TRUE) == 
      "") {
      result = as.data.frame(matrix("", ncol = length(attributes), 
        nrow = 0), stringsAsFactors = FALSE)
    }
    else {
      if (length(grep("^Query ERROR", postRes)) > 0L) 
        stop(postRes)
      con = textConnection(postRes)
      result = read.table(con, sep = "\t", header = callHeader, 
        quote = quote, comment.char = "", check.names = FALSE, 
        stringsAsFactors = FALSE)
      if (verbose) {
        writeLines("#################\nParsed results:")
        print(result)
      }
      close(con)
      if (!(is(result, "data.frame") && (ncol(result) == 
        length(attributes)))) {
        print(head(result))
        stop("The query to the BioMart webservice returned an invalid result: the number of columns in the result table does not equal the number of attributes in the query. \nPlease report this on the support site at http://support.bioconductor.org")
      }
    }
    resultList[[i]] <- .setResultColNames(result, mart = mart, 
      attributes = attributes, bmHeader = bmHeader)
  }
  result <- do.call("rbind", resultList)
  return(result)
}

I really appreciate any help in this matter.

ADD REPLYlink modified 8 weeks ago by Martin Morgan ♦♦ 23k • written 8 weeks ago by Oskar50

Ensembl's BioMart seems to be exceptionally slow for me at the moment - I suspect you're encountering the same problem as before, but even our smaller windows are timing out.

Maybe you can try the REST API instead. The following code works similarly to before, but uses the REST API rather than BioMart to retrieve the data, and seem much faster to me.

library(httr)
library(jsonlite)

chr <- 8
s1 <- seq(143262170, 144261715, by = 10000)
s2 <- c(s1[-1]-1, 144261715)
regions <- matrix(c(rep(chr, length(s1)), s1, s2), ncol = 3)

get_snps_via_rest <- function(values) {
  server <- "https://grch37.rest.ensembl.org"
  ext <- paste0("/overlap/region/human/", values[1], ":", values[2], "-", values[3], "?feature=variation;")
  r <- GET(paste(server, ext, sep = ""), content_type("application/json"))
  fromJSON(content(r, as = "text", encoding = "UTF-8"))
}

res_list2 <- apply(regions, 1, get_snps_via_rest)
snp_id8 <- do.call(rbind, res_list2)

You don't have quite the same control over the output columns, but I think everything you want should still be present:

>  head(snp_id8)
  alleles   consequence_type source           id strand       end assembly_name feature_type     start clinical_significance seq_region_name
1    C, T intergenic_variant  dbSNP  rs139849327      1 143262170        GRCh37    variation 143262170                  NULL               8
2    G, C intergenic_variant  dbSNP rs1337146651      1 143262177        GRCh37    variation 143262177                  NULL               8
3 G, A, T intergenic_variant  dbSNP  rs549325531      1 143262179        GRCh37    variation 143262179                  NULL               8
4    G, A intergenic_variant  dbSNP rs1222699542      1 143262183        GRCh37    variation 143262183                  NULL               8
5    T, A intergenic_variant  dbSNP  rs896634493      1 143262190        GRCh37    variation 143262190                  NULL               8
6    G, A intergenic_variant  dbSNP  rs187036709      1 143262193        GRCh37    variation 143262193                  NULL               8
ADD REPLYlink written 8 weeks ago by Mike Smith3.3k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 103 users visited in the last hour