Minimal group sizes in permutation tests
2
0
Entering edit mode
Benjamin Otto ▴ 830
@benjamin-otto-1519
Last seen 8.2 years ago
Hey all, 1. For statistical tests there are usually minimal group sizes recommended for appropriate working. For a chi-square test as example the lower level was 10 obersvations in each field of the table, if I remember correctly. What about permutation tests? Is there some kind of minimal recommendation for group sizes? I can't find any hint on that. 2. As far as I understand the permutation p-value is given by the quantile describing the position of the native p-value in the permutation p-value distribution. So for 100 permutations and 5 values smaller than the native one the new p-value would be 0.05. What happens when the original p-value is the absolut minimum? Is such a thing like p-value equals zero defined? 3. Given a design of 3x3 samples (20 permutations), will the test return reasonable values? Doesn't look like it to me. Best regards, Benjamin Otto ====================================== Benjamin Otto University Hospital Hamburg-Eppendorf Institute For Clinical Chemistry Martinistr. 52 D-20246 Hamburg Tel.: +49 40 42803 1908 Fax.: +49 40 42803 4971 ====================================== -- Pflichtangaben gem?? Gesetz ?ber elektronische Handelsregister und Genossenschaftsregister sowie das Unternehmensregister (EHUG): Universit?tsklinikum Hamburg-Eppendorf K?rperschaft des ?ffentlichen Rechts Gerichtsstand: Hamburg Vorstandsmitglieder: Prof. Dr. J?rg F. Debatin (Vorsitzender) Dr. Alexander Kirstein Ricarda Klein Prof. Dr. Dr. Uwe Koch-Gromus
• 1.5k views
ADD COMMENT
0
Entering edit mode
@james-w-macdonald-5106
Last seen 17 minutes ago
United States
Hi Benjamin, Benjamin Otto wrote: > Hey all, > > 1. For statistical tests there are usually minimal group sizes recommended > for appropriate working. For a chi-square test as example the lower level > was 10 obersvations in each field of the table, if I remember correctly. > What about permutation tests? Is there some kind of minimal recommendation > for group sizes? I can't find any hint on that. The lowest p-value you can get (aside from 0) will be 1/n, where n == # of combinations. The usual recommendation for any sort of permutation/sampling test (bootstrap, jackknife, etc) is 500 - 1000 permutations, so you need at least 6 samples per group if you want to follow that recommendation. > > 2. As far as I understand the permutation p-value is given by the quantile > describing the position of the native p-value in the permutation p-value > distribution. So for 100 permutations and 5 values smaller than the native > one the new p-value would be 0.05. What happens when the original p-value is > the absolut minimum? Is such a thing like p-value equals zero defined? Sure. If the observed statistic is larger than _all_ the permuted statistics, then the permuted p-value is 0/n. Of course if your permuted null is really coarse (say only 20 combinations) then this is less impressive than if you had much more. > > 3. Given a design of 3x3 samples (20 permutations), will the test return > reasonable values? Doesn't look like it to me. Well your permuted null will be very coarse. IMO, in this case using standard normal theory is preferable. Best, Jim > > Best regards, > > Benjamin Otto > > > ====================================== > Benjamin Otto > University Hospital Hamburg-Eppendorf > Institute For Clinical Chemistry > Martinistr. 52 > D-20246 Hamburg > > Tel.: +49 40 42803 1908 > Fax.: +49 40 42803 4971 > ====================================== > > > > > > -------------------------------------------------------------------- ---- > > _______________________________________________ > Bioconductor mailing list > Bioconductor at stat.math.ethz.ch > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor -- James W. MacDonald, M.S. Biostatistician Affymetrix and cDNA Microarray Core University of Michigan Cancer Center 1500 E. Medical Center Drive 7410 CCGC Ann Arbor MI 48109 734-647-5623
ADD COMMENT
0
Entering edit mode
Claus Mayer ▴ 340
@claus-mayer-1179
Last seen 8.2 years ago
European Union
Hi Benjamin, Benjamin Otto wrote: > Hey all, > > 1. For statistical tests there are usually minimal group sizes recommended > for appropriate working. For a chi-square test as example the lower level > was 10 obersvations in each field of the table, if I remember correctly. > What about permutation tests? Is there some kind of minimal recommendation > for group sizes? I can't find any hint on that. > The group sizes determine how many different possible permutations they are, eg. with 3 samples each in 2 groups you only have 20 permutations. If you would use a 1-sided permutation-test in that situation the smallest possible p-value thus would be 5%, i.e. you have no chance to ever find a significant result at a 5% level (for a 2-sided test you wouldn't even be able to get below 10%). In my opinion the number of different permutations should be at least in the hundreds, so for a 2-group comparison I wouldn't use a permutation test for anything less than 5 per group (in which case you have 252 permutations). For other designs you would have to calculate the number of possible permutations to see whether it makes sense. Apart from that I see little other constraints in using a permutation test as long as you are sure that under the nullhypothesis you are testing the variables are "i.i.d" (= independently indentically distributed) > 2. As far as I understand the permutation p-value is given by the quantile > describing the position of the native p-value in the permutation p-value > distribution. So for 100 permutations and 5 values smaller than the native > one the new p-value would be 0.05. What happens when the original p-value is > the absolut minimum? Is such a thing like p-value equals zero defined? > The classical definition of the p-value calculates the probability of observing an outcome as extrem or even more extrem as the observed one. So if the observed value is the smalles of the 100 permutations your p-value would 1%. > 3. Given a design of 3x3 samples (20 permutations), will the test return > reasonable values? Doesn't look like it to me. > for 2x3 samples it would be 20 permutations, but for 3x3 the number will be bigger Claus > Best regards, > > Benjamin Otto > > > ====================================== > Benjamin Otto > University Hospital Hamburg-Eppendorf > Institute For Clinical Chemistry > Martinistr. 52 > D-20246 Hamburg > > Tel.: +49 40 42803 1908 > Fax.: +49 40 42803 4971 > ====================================== > > > > > -------------------------------------------------------------------- ---- > > _______________________________________________ > Bioconductor mailing list > Bioconductor at stat.math.ethz.ch > https://stat.ethz.ch/mailman/listinfo/bioconductor > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor > -------------------------------------------------------------------- ---- > > > -- ********************************************************************** ************* Dr Claus-D. Mayer | http://www.bioss.ac.uk Biomathematics & Statistics Scotland | email: claus at bioss.ac.uk Rowett Research Institute | Telephone: +44 (0) 1224 716652 Aberdeen AB21 9SB, Scotland, UK. | Fax: +44 (0) 1224 715349
ADD COMMENT
0
Entering edit mode
Hi, Claus-Dieter Mayer wrote: > Hi Benjamin, > > Benjamin Otto wrote: >> Hey all, >> >> 1. For statistical tests there are usually minimal group sizes recommended >> for appropriate working. For a chi-square test as example the lower level >> was 10 obersvations in each field of the table, if I remember correctly. >> What about permutation tests? Is there some kind of minimal recommendation >> for group sizes? I can't find any hint on that. >> > The group sizes determine how many different possible permutations they > are, eg. with 3 samples each in 2 groups you only have 20 permutations. > If you would use a 1-sided permutation-test in that situation the > smallest possible p-value thus would be 5%, i.e. you have no chance to > ever find a significant result at a 5% level (for a 2-sided test you > wouldn't even be able to get below 10%). In my opinion the number of > different permutations should be at least in the hundreds, so for a > 2-group comparison I wouldn't use a permutation test for anything less > than 5 per group (in which case you have 252 permutations). For other > designs you would have to calculate the number of possible permutations > to see whether it makes sense. > Apart from that I see little other constraints in using a permutation > test as long as you are sure that under the nullhypothesis you are > testing the variables are "i.i.d" (= independently indentically distributed) Unless you have more possible permutations than you can compute, please make sure that you just enumerate all permutations, compute the test statistic for each and use that for your reference distribution. It makes no sense to sample from this distribution by generating random permutations if the number of permutations is small. >> 2. As far as I understand the permutation p-value is given by the quantile >> describing the position of the native p-value in the permutation p-value >> distribution. So for 100 permutations and 5 values smaller than the native >> one the new p-value would be 0.05. What happens when the original p-value is >> the absolut minimum? Is such a thing like p-value equals zero defined? >> > The classical definition of the p-value calculates the probability of > observing an outcome as extrem or even more extrem as the observed one. > So if the observed value is the smalles of the 100 permutations your > p-value would 1%. >> 3. Given a design of 3x3 samples (20 permutations), will the test return >> reasonable values? Doesn't look like it to me. >> > for 2x3 samples it would be 20 permutations, but for 3x3 the number will > be bigger > > Claus >> Best regards, >> >> Benjamin Otto >> >> >> ====================================== >> Benjamin Otto >> University Hospital Hamburg-Eppendorf >> Institute For Clinical Chemistry >> Martinistr. 52 >> D-20246 Hamburg >> >> Tel.: +49 40 42803 1908 >> Fax.: +49 40 42803 4971 >> ====================================== >> >> >> >> >> ------------------------------------------------------------------- ----- >> >> _______________________________________________ >> Bioconductor mailing list >> Bioconductor at stat.math.ethz.ch >> https://stat.ethz.ch/mailman/listinfo/bioconductor >> Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor >> ------------------------------------------------------------------- ----- >> >> >> > -- Robert Gentleman, PhD Program in Computational Biology Division of Public Health Sciences Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N, M2-B876 PO Box 19024 Seattle, Washington 98109-1024 206-667-7700 rgentlem at fhcrc.org
ADD REPLY

Login before adding your answer.

Traffic: 548 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6