Dear friends, I have been trying to perform ROC analysis with ROCR. Since there is not much support for the queries I have switched to ROC package of Bioconductor. I'm trying to perform ROC analysis on Methylation data obtained from MeDiP experiment.The data set has values ranging from 0 to as large as 5. I have a couple of doubts with ROC as I had with ROCR. 1. When provided with different threshold values such as 0.6,0.7,0.8,0.9,each time I've got a plot which has the same curve as shown in the attached pdf.There's no change whatsoever in the curve with the different thresholds applied.Is the result what i'm getting on the data set appropriate?(I've come across research papers with ROC analysis being implemented on Methylation data) 2. As ROC provides knowldege about the cut-off value for micro array data,while assigning a cut-off value,should one take into account the value of threshold given in ROC or the accuracy value? Following is my sessional info. load("RGKma.RData") state <-ifelse(RGKma$M[1:100,3] > 0.9, 1,0) print("RGKma$M:");print(RGKma$M[1:100,3]) print("state:");print(state) data<-RGKma$M[1:100,3] R1<-rocdemo.sca(truth=state,data,dxrule.sca) pdf("rocK.pdf") plot(R1,col = "red") dev.off() print("ROC(R1):");print(ROC(R1,.3)) [1] "RGKma$M:" ? [1]? 2.10538709 -0.07335174? 2.13920582? 0.18499421? 3.30846203? 1.69065450 ? [7]? 4.24969667? 1.37415619? 1.65769067? 5.39253767? 1.19349192? 5.40321575 ?[13]? 3.06468274? 1.34311072? 0.68093156? 4.03579639? 2.91909842? 3.36384055 ?[19]? 3.54968030? 4.06977722? 2.31968962? 3.17237025? 2.80040216? 3.01874372 ?[25]? 1.89894809? 4.17251372 -0.92690849? 2.72505883? 1.10609889? 2.33584882 ?[31]? 0.09886450? 3.30066347? 2.66466248? 1.39238431? 2.38782229? 4.19572478 ?[37]? 3.97185357? 0.38627851 -0.09439237 -0.22948185? 3.45955944? 0.64538744 ?[43]? 1.02627932 -0.53789425? 4.17758537? 2.87612185? 3.25867248? 1.89058878 ?[49]? 2.71612450? 3.06751911? 2.63941028? 1.03250743? 2.07739372 -0.11727572 ?[55]? 3.66338130? 2.52249841? 0.05683122? 1.90834958? 4.25784185? 1.87577855 ?[61]? 0.21814006? 0.98911168? 1.63475517? 4.57600122? 0.99326629? 1.86706117 ?[67]? 1.27215099? 2.23056201 -0.81404957? 1.12010588? 1.62733217? 0.41223049 ?[73]? 3.43584658? 3.78533569? 2.33141286? 3.15227631? 1.51317488? 3.37017353 ?[79] -0.57605695? 2.96351684? 2.82082253? 2.85149236? 1.43692942 -0.49898928 ?[85] -0.81504931 -0.75064053? 1.11314716? 2.51744122? 2.49526189 -1.17086212 ?[91]? 1.11677841? 0.51370382? 3.24834409? 0.40958307? 0.39834589? 1.28139084 ?[97]? 1.24613108? 3.91323816? 2.06097801? 2.88980181 [1] "state:" ? [1] 1 0 1 0 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0 1 1 1 1 1 1 ?[38] 0 0 0 1 0 1 0 1 1 1 1 1 1 1 1 1 0 1 1 0 1 1 1 0 1 1 1 1 1 1 1 0 1 1 0 1 1 ?[75] 1 1 1 1 0 1 1 1 1 0 0 0 1 1 1 0 1 0 1 0 0 1 1 1 1 1 [1] "ROC(R1):" [1] 1 Thanking you in anticipation, Susan, M.Sc. in Molecular Biology and Human Genetics, Manipal Life Sciences Centre, Manipal,India. The INTERNET now has a personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: rocK.pdf Type: application/pdf Size: 4283 bytes Desc: not available URL: <https: stat.ethz.ch="" pipermail="" bioconductor="" attachments="" 20100119="" 8a17266a="" attachment.pdf="">

On Tue, Jan 19, 2010 at 1:51 AM, Susan Bosco <susanbosco86 at="" yahoo.com=""> wrote: > Dear friends, > > I have been trying to perform ROC analysis with ROCR. Since there is not much support for the queries I have switched to ROC package of Bioconductor. > > I'm trying to perform ROC analysis on Methylation data obtained from MeDiP experiment.The data set has values ranging from 0 to as large as 5. > > I have a couple of doubts with ROC as I had with ROCR. > 1. When provided with different threshold values such as 0.6,0.7,0.8,0.9,each time I've got a plot which has the same curve as shown in the attached pdf.There's no change whatsoever in the curve with the different thresholds applied.Is the result > ?what i'm getting on the data set appropriate?(I've come across research papers with ROC analysis being implemented on Methylation data) > > 2. As ROC provides knowldege about the cut-off value for micro array data,while assigning a cut-off value,should one take into account the value of threshold given in ROC or the accuracy value? > > Following is my sessional info. > > load("RGKma.RData") > state <-ifelse(RGKma$M[1:100,3] > 0.9, 1,0) > print("RGKma$M:");print(RGKma$M[1:100,3]) > print("state:");print(state) > data<-RGKma$M[1:100,3] > R1<-rocdemo.sca(truth=state,data,dxrule.sca) > pdf("rocK.pdf") > plot(R1,col = "red") > dev.off() > print("ROC(R1):");print(ROC(R1,.3)) > > [1] "RGKma$M:" > ? [1]? 2.10538709 -0.07335174? 2.13920582? 0.18499421? 3.30846203? 1.69065450 > ? [7]? 4.24969667? 1.37415619? 1.65769067? 5.39253767? 1.19349192? 5.40321575 > ?[13]? 3.06468274? 1.34311072? 0.68093156? 4.03579639? 2.91909842? 3.36384055 > ?[19]? 3.54968030? 4.06977722? 2.31968962? 3.17237025? 2.80040216? 3.01874372 > ?[25]? 1.89894809? 4.17251372 -0.92690849? 2.72505883? 1.10609889? 2.33584882 > ?[31]? 0.09886450? 3.30066347? 2.66466248? 1.39238431? 2.38782229? 4.19572478 > ?[37]? 3.97185357? 0.38627851 -0.09439237 -0.22948185? 3.45955944? 0.64538744 > ?[43]? 1.02627932 -0.53789425? 4.17758537? 2.87612185? 3.25867248? 1.89058878 > ?[49]? 2.71612450? 3.06751911? 2.63941028? 1.03250743? 2.07739372 -0.11727572 > ?[55]? 3.66338130? 2.52249841? 0.05683122? 1.90834958? 4.25784185? 1.87577855 > ?[61]? 0.21814006? 0.98911168? 1.63475517? 4.57600122? 0.99326629? 1.86706117 > ?[67]? 1.27215099? 2.23056201 -0.81404957? 1.12010588? 1.62733217? 0.41223049 > ?[73]? 3.43584658? 3.78533569? 2.33141286? 3.15227631? 1.51317488? 3.37017353 > ?[79] -0.57605695? 2.96351684? 2.82082253? 2.85149236? 1.43692942 -0.49898928 > ?[85] -0.81504931 -0.75064053? 1.11314716? 2.51744122? 2.49526189 -1.17086212 > ?[91]? 1.11677841? 0.51370382? 3.24834409? 0.40958307? 0.39834589? 1.28139084 > ?[97]? 1.24613108? 3.91323816? 2.06097801? 2.88980181 > [1] "state:" > ? [1] 1 0 1 0 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0 1 1 1 1 1 1 > ?[38] 0 0 0 1 0 1 0 1 > ?1 1 1 1 1 1 1 1 0 1 1 0 1 1 1 0 1 1 1 1 1 1 1 0 1 1 0 1 1 > ?[75] 1 1 1 1 0 1 1 1 1 0 0 0 1 1 1 0 1 0 1 0 0 1 1 1 1 1 > [1] "ROC(R1):" > [1] 1 Hi, Susan. Your code is fine and the ROC package is doing as advertised, I think. There may be a bit of a misunderstanding of what an ROC curve represents. An ROC curve presents the False Positive Rate versus the True Positive Rate across the thresholds in the data (that is, ALL thresholds). It can be used to pick the best threshold from all possible thresholds. Try your plot function like so: plot(R1,show.thresh=TRUE) For your data, if the threshold is below 0.9, you will never see a false positive in your data and the true positive rate will simply increase, resulting in the vertical line at x=0. As soon as the threshold passes 0.9, you have captured all the true positives and everything else is a false positive, resulting in the rest of the plot, a horizontal line at y=1. Try this: state[1:10]=1 and do the plot again. Since the data are now not perfectly described by a threshold rule (there is some noise), you will see a different plot. Let us know if there are other questions. Sean

Dear Sean, Thank you so much for  the help. I tried with a range of thresholds from 0-0.9..As you had mentioned,the true positive rates no doubt increased with thresholds below 0.9.However I did get some false positive rates even at a minimum threshold of 0.1.Could you kindly explain the reason? Is there any method of finding the optimal threshold,maximizing the true positive rates while minimizing the false positives,instead of randomly choosing between 0-0.9? Thanks in advance, Susan. The INTERNET now has a personality. YOURS! See your Yahoo! Homepage. [[alternative HTML version deleted]]

Dear Sean, Thanks again. I corrected the script changing the value of 'truth' variable with rbinom() function. Since my data size is quite large(data is of 244K),I tried with the first 200,for which I was able to find proper ROC curve. However, when I include the complete data, the plot changes. For the whole data,I get a linear graph with small variations. My sessionInfo() looks like this: For 100 values of the data: library(ROC) load("RGKma.RData") state= rbinom(length(RGKma$M[1:100,3]),1,0.33) data = RGKma$M[1:200,3] R1<-rocdemo.sca(truth=state,data,dxrule.sca) pdf("ROCk.pdf") plot(R1, show.thresh=TRUE,col = "red") dev.off() For the complete data: library(ROC) load("RGKma.RData") state= rbinom(length(RGKma$M[,3]),1,0.33) data = RGKma$M[,3] R1<-rocdemo.sca(truth=state,data,dxrule.sca) pdf("ROCallk.pdf") plot(R1, show.thresh=TRUE,col = "red") dev.off() I've hereby attached the pdfs of the plots.I would appreciate if you could help me out with this problem that I encountered with a large data size. Thanking you sincerely, Susan. --- On Wed, 20/1/10, Sean Davis <seandavi at="" gmail.com=""> wrote: From: Sean Davis <seandavi at="" gmail.com=""> Subject: Re: [BioC] Seeking assistance on ROC To: "Susan Bosco" <susanbosco86 at="" yahoo.com=""> Cc: bioconductor at stat.math.ethz.ch, "prashantha hebbar" <prashantha.hebbar at="" manipal.edu=""> Date: Wednesday, 20 January, 2010, 12:05 PM On Wed, Jan 20, 2010 at 12:39 AM, Susan Bosco <susanbosco86 at="" yahoo.com=""> wrote: Dear Sean, Thank you so much for? the help. I tried with a range of thresholds from 0-0.9..As you had mentioned,the true positive rates no doubt increased with thresholds below 0.9.However I did get some false positive rates even at a minimum threshold of 0.1.Could you kindly explain the reason? Is there any method of finding the optimal threshold,maximizing the true positive rates while minimizing the false positives,instead of randomly choosing between 0-0.9? Hi, Susan.? The ROC curve IS that method.? The ROC curve represents ALL thresholds as applied to the data.? If you plot with show.thresh=TRUE, you will see the thresholds that were tried and where they are on the curve.? If the threshold to which you are referring is the one that you used to determine the variable you called "state", then we are talking about two different things.? The "truth" variable is meant to be assigned by some source other than the data themselves.? If you do not know the true state of your samples and find yourself assigning the state the data, then ROC curve analysis will not be of any use. Sean ? Thanks in advance, Susan. The INTERNET now has a personality. YOURS! See your Yahoo! Homepage. Your Mail works best with the New Yahoo Optimized IE8. Get it NOW! http://downloads.yahoo.com/in/internetexplorer/ -------------- next part -------------- A non-text attachment was scrubbed... Name: ROC_K.pdf Type: application/pdf Size: 10758 bytes Desc: not available URL: <https: stat.ethz.ch="" pipermail="" bioconductor="" attachments="" 20100123="" 5dcd5abd="" attachment-0002.pdf=""> -------------- next part -------------- A non-text attachment was scrubbed... Name: ROC_allK.pdf Type: application/pdf Size: 17944800 bytes Desc: not available URL: <https: stat.ethz.ch="" pipermail="" bioconductor="" attachments="" 20100123="" 5dcd5abd="" attachment-0003.pdf="">