Question: Zero values for bead backgrounds (reading idat files using Illuminaio)
1
4.0 years ago by
willj30
France
willj30 wrote:

I have a batch of idat files that I would like to analyze using Immuminaio and lumi. This is the first time I have analyzed raw Illumina idat files and I have used neither Immuminaio nor lumi before. The problem I'll describe is not necessarily related to Illuminiao - it may be a problem with my idat files themselves.

If I read in the example data following the commands in the illuminaio.pdf vignette, the output data look correct, but when I read in one of my own idat files, the columns BackgroundBinData and BackgroundDevBinData have all values at '0'.

Example data:

> head(idatData_example$BackgroundBinData) [1] 657.5565 657.1952 657.0317 657.4821 657.4556 657.2791 > head(idatData_example$BackgroundDevBinData)
[1] 1.130837 1.345547 1.438468 1.250080 1.411696 1.081928

My data:

> head(idatData$BackgroundBinData) [1] 0 0 0 0 0 0 > head(idatData$BackgroundDevBinData)
[1] 0 0 0 0 0 0

One of my idat files can be downloaded here: https://dl.dropboxusercontent.com/u/61432493/3998479036_A_Grn.idat

My questions are:

- Why are my idat files giving these 0 values?  Has there been a problem during the data generation? (I no longer have access to any raw image data)

- Do I need these columns in order to proceed to downstream analyses in Lumi?

I have already tried running the batch through Lumi using some code given at Illumina Expression analysis from iDAT files, and I have encountered an issue there, but I don't know if this second issue is related to the problem I've just described. I can give more details if needed but, in brief: whereas the example.lumi batch that can be generated by following the lumi vignette contains the following element names: "beadNum, detection, exprs, se.exprs", but my own batch is missing the detection column - it contains only element names: "beadNum, exprs, se.exprs".

This seems to cause a problem when I try to do VST variance stabilizing transform. The following error is given:

> x.T <- lumiT(x)
Perform vst transformation ...
Error in !assayDataValidMembers(assayData(x.lumi), "detection") :
invalid argument type

Many thanks for any help.

beadarray illuminaio idat • 820 views
modified 4.0 years ago by Mike Smith3.9k • written 4.0 years ago by willj30
Answer: Zero values for bead backgrounds (reading idat files using Illuminaio)
1
4.0 years ago by
United States
Kasper Daniel Hansen6.4k wrote:
Illumina IDAT files are different in what data is contained within them. To me, it seems that the background values are set to 0; this is not likely to happen by an error. I don't know what fields lumi might assume are present. Best, Kasper On Mon, Sep 7, 2015 at 9:19 AM, willj [bioc] <noreply@bioconductor.org> wrote: > Activity on a post you are following on support.bioconductor.org > > User willj <https: support.bioconductor.org="" u="" 8763=""/> wrote Question: > Zero values for bead backgrounds (reading idat files using Illuminaio) > <https: support.bioconductor.org="" p="" 71887=""/>: > > I have a batch of idat files that I would like to analyze using Immuminaio > and lumi. This is the first time I have analyzed raw Illumina idat files > and I have used neither Immuminaio nor lumi before. The problem I'll > describe is not necessarily related to Illuminiao - it may be a problem > with my idat files themselves. > > If I read in the example data following the commands in the illuminaio.pdf > vignette, the output data look correct, but when I read in one of my own > idat files, the columns BackgroundBinData and BackgroundDevBinData have all > values at '0'. > > > > Example data: > > > head(idatData_example$BackgroundBinData) > [1] 657.5565 657.1952 657.0317 657.4821 657.4556 657.2791 > > > head(idatData_example$BackgroundDevBinData) > [1] 1.130837 1.345547 1.438468 1.250080 1.411696 1.081928 > > > > My data: > > > head(idatData$BackgroundBinData) > [1] 0 0 0 0 0 0 > > head(idatData$BackgroundDevBinData) > [1] 0 0 0 0 0 0 > > > > One of my idat files can be downloaded here: > https://dl.dropboxusercontent.com/u/61432493/3998479036_A_Grn.idat > > > > My questions are: > > - Why are my idat files giving these 0 values? Has there been a problem > during the data generation? (I no longer have access to any raw image data) > > - Do I need these columns in order to proceed to downstream analyses in > Lumi? > > I have already tried running the batch through Lumi using some code given > at Illumina Expression analysis from iDAT files > <https: support.bioconductor.org="" p="" 56248=""/>, and I have encountered an > issue there, but I don't know if this second issue is related to the > problem I've just described. I can give more details if needed but, in > brief: whereas the example.lumi batch that can be generated by following > the lumi vignette contains the following element names: "beadNum, > detection, exprs, se.exprs", but my own batch is missing the detection > column - it contains only element names: "beadNum, exprs, se.exprs". > > This seems to cause a problem when I try to do VST variance stabilizing > transform. The following error is given: > > > x.T <- lumiT(x) > Perform vst transformation ... > Error in !assayDataValidMembers(assayData(x.lumi), "detection") : > invalid argument type > > > > Many thanks for any help. > > > > ------------------------------ > > Post tags: illuminaio, idat, beadarray > > You may reply via email or visit Zero values for bead backgrounds (reading idat files using Illuminaio) >
Answer: Zero values for bead backgrounds (reading idat files using Illuminaio)
1
4.0 years ago by
United Kingdom
andrew.j.skelton73320 wrote:

Gene Expression data: you can use the lumidat package (github) to read your IDAT files and create a sample probe profile. - I'm going to assume it's gene expression as you mention VST transformations. (alternatively, use GenomeStudio, or the array company should have sent you a copy of the sample probe profile.)

Thanks Andrew, lumidat works very well. One question I have though: on comparing (for the same idats) output of Genomestudio with output of lumidat, I noticed that the "detection" value given by lumidat is the inverse (i.e. 1-x) of the "detection_pval" given by GenomeStudio. Why is that?

I haven't noticed any differences between detection P values between the two methods - it could be a parameter issue? Something extra ran in GenomeStudio. If you do more testing and can't determine the nature of the difference, it might be worth writing a comment on the lumidat github page.

I've ran several idat files now through lumidat, including the example idat files, and the "Detection" column always has it's highest values (close to 1) for the probes having the strongest signal strength. I have Genome Studio output for some of these idats. There are minimal changes in the ranking of probes according to signal strength but the Genome Studio "Detection Pval" for the strongest probes is usually close to 0. It makes sense to me that the probes with the strongest signal would be the most confidently detected and thus have the lowest detection p-values. So, maybe the lumidat "Detection" column is not intended to represent the detection p-value? Maybe it represents detection type 2 error?

ADD REPLYlink modified 4.0 years ago • written 4.0 years ago by willj30

Does lumidat gives you non-zero backgrounds, or is the "raw" representation of an idat file the same?  Otherwise we should look into fixing illuminaio

ADD REPLYlink written 4.0 years ago by Kasper Daniel Hansen6.4k

Hi Kasper - there is no problem with Illuminaio (both packages give me those zero values). However, lumidat has the advantage that it calculates a "detection" value, and feeds easily into Lumi. My main problem was getting from Illuminaio to Lumi. I still don't know why my idats have those zero values, but perhaps they are not important to the downstream analysis. (Sorry if the thread became a bit confusing - Andrew's advice on lumidat helped me greatly to go forwards with the analysis, but it doesn't address those zero values that were the main subject of my first post.)

ADD REPLYlink written 4.0 years ago by Kasper Daniel Hansen6.4k
Answer: Zero values for bead backgrounds (reading idat files using Illuminaio)
1
4.0 years ago by
Mike Smith3.9k
EMBL Heidelberg / de.NBI
Mike Smith3.9k wrote:

Just to follow this up, I've seen both real values and zeros stored as background in various idat files that I've looked at. However, I've never been sure that any sort of bead-type specific background value is useful when dealing with summarised data.  When dealing with bead-level data the intensity of a probe may be affected by some local phenomena, so taking the local background into account may make sense.  Once you've summarised values from beads that are hopefully spread evenly over the array surface, to get the values in an idat, I'm not sure there's anything meaningful to be gained from also summarising and storing background values.

I suspect Illumina came to a similar conclusion and turned off storing it, hence the difference in the files.  I've never seen any mention of bead-summary background when using GenomeStudio to look at idat files.