Question: IHW scale of problem
1
gravatar for xiaotong.yao23
10 weeks ago by
xiaotong.yao2310 wrote:

Hi,

I'm trying to use IHW package to achieve covariate weighted p-values for about 35 million hypothesis testings, but the function has been running for 5 hours and still hasn't returned or thrown an error. I'm wondering if 35 million is stretching its scale and what would be the maximal amount of hypothesis it could reasonably handle. Besides, are there any way to speed it up?

Thanks, Xiaotong

Command

pval.nb.ihw.refd = ihw(rand.p ~ ref.d, data = pval.nb, alpha = 0.05)

nrow(pval.nb) [1] 353793300

ADD COMMENTlink modified 9 weeks ago by Wolfgang Huber13k • written 10 weeks ago by xiaotong.yao2310
Answer: IHW scale of problem
0
gravatar for Nikos Ignatiadis
10 weeks ago by
Heidelberg
Nikos Ignatiadis160 wrote:

Hi Xiaotong,

For such large problems one can get large speed savings by doing computations on a subset of all p-values [for example, p-values >= 0.01 are very unlikely to be rejected for a problem of such scale, but at the same time, working with p-values <=0.01 effectively reduces the number of p-values by 2 orders of magnitude]. However, this subsetting has to be done carefully. Please read Section 4 in the IHW vignette ("Advanced usage: Working with incomplete p-value lists") for an explanation of how this may be achieved.

Hope this helps! Nikos

ADD COMMENTlink written 10 weeks ago by Nikos Ignatiadis160
Answer: IHW scale of problem
0
gravatar for Wolfgang Huber
9 weeks ago by
EMBL European Molecular Biology Laboratory
Wolfgang Huber13k wrote:

In addition, you can randomly sample your hypotheses with various sample sizes, measure run time using e.g. system.time or the microbenchmark package, plot time against number of hypotheses, see that the relationship is linear, measure the slope, and predict the run time for your problem size. Nikos' solution is of course more elegant.

ADD COMMENTlink written 9 weeks ago by Wolfgang Huber13k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 239 users visited in the last hour