I have a batch of idat files that I would like to analyze using Immuminaio and lumi. This is the first time I have analyzed raw Illumina idat files and I have used neither Immuminaio nor lumi before. The problem I'll describe is not necessarily related to Illuminiao - it may be a problem with my idat files themselves.
If I read in the example data following the commands in the illuminaio.pdf vignette, the output data look correct, but when I read in one of my own idat files, the columns BackgroundBinData and BackgroundDevBinData have all values at '0'.
Example data:
> head(idatData_example$BackgroundBinData)
[1] 657.5565 657.1952 657.0317 657.4821 657.4556 657.2791
> head(idatData_example$BackgroundDevBinData)
[1] 1.130837 1.345547 1.438468 1.250080 1.411696 1.081928
My data:
> head(idatData$BackgroundBinData)
[1] 0 0 0 0 0 0
> head(idatData$BackgroundDevBinData)
[1] 0 0 0 0 0 0
One of my idat files can be downloaded here: https://dl.dropboxusercontent.com/u/61432493/3998479036_A_Grn.idat
My questions are:
- Why are my idat files giving these 0 values? Has there been a problem during the data generation? (I no longer have access to any raw image data)
- Do I need these columns in order to proceed to downstream analyses in Lumi?
I have already tried running the batch through Lumi using some code given at Illumina Expression analysis from iDAT files, and I have encountered an issue there, but I don't know if this second issue is related to the problem I've just described. I can give more details if needed but, in brief: whereas the example.lumi batch that can be generated by following the lumi vignette contains the following element names: "beadNum, detection, exprs, se.exprs", but my own batch is missing the detection column - it contains only element names: "beadNum, exprs, se.exprs".
This seems to cause a problem when I try to do VST variance stabilizing transform. The following error is given:
> x.T <- lumiT(x)
Perform vst transformation ...
Error in !assayDataValidMembers(assayData(x.lumi), "detection") :
invalid argument type
Many thanks for any help.
Thanks Andrew, lumidat works very well. One question I have though: on comparing (for the same idats) output of Genomestudio with output of lumidat, I noticed that the "detection" value given by lumidat is the inverse (i.e. 1-x) of the "detection_pval" given by GenomeStudio. Why is that?
I haven't noticed any differences between detection P values between the two methods - it could be a parameter issue? Something extra ran in GenomeStudio. If you do more testing and can't determine the nature of the difference, it might be worth writing a comment on the lumidat github page.
I've ran several idat files now through lumidat, including the example idat files, and the "Detection" column always has it's highest values (close to 1) for the probes having the strongest signal strength. I have Genome Studio output for some of these idats. There are minimal changes in the ranking of probes according to signal strength but the Genome Studio "Detection Pval" for the strongest probes is usually close to 0. It makes sense to me that the probes with the strongest signal would be the most confidently detected and thus have the lowest detection p-values. So, maybe the lumidat "Detection" column is not intended to represent the detection p-value? Maybe it represents detection type 2 error?
Does lumidat gives you non-zero backgrounds, or is the "raw" representation of an idat file the same? Otherwise we should look into fixing illuminaio
Hi Kasper - there is no problem with Illuminaio (both packages give me those zero values). However, lumidat has the advantage that it calculates a "detection" value, and feeds easily into Lumi. My main problem was getting from Illuminaio to Lumi. I still don't know why my idats have those zero values, but perhaps they are not important to the downstream analysis. (Sorry if the thread became a bit confusing - Andrew's advice on lumidat helped me greatly to go forwards with the analysis, but it doesn't address those zero values that were the main subject of my first post.)