Two POINTS:
First, I think people are missing the point about statistics. When
you
look at tens of thousands of genes, random chance plays a BIG role.
This means that large fold changes will occur and very small p-values
will occur often by chance alone. The investigator doesn't know which
large fold changes or small p-values are due to chance or which are
"true" results.
The best we can do is to try to apply statisics (and biology) with
these
things in mind, that's what statistics is all about (one of the things
anyway),i.e. applying methods in such a way as to get results that are
meaningful and for which the probabilities of making different types
of
errors are known.
With respect to just looking at the rankings, with small samples the
distributions are well known, very discrete and have very few
distinct
probabilities. The situation that Mike describes, i.e. 10 samples
with
5 per group is a situation for which there are 252 possible outcomes
(10
choose 5 combinatorically). If you consider both up and down
regulation
as an outcome of interest there are 126 distinct outcomes so that the
probability of this occuring by chance alone is 1 in 126 or .00794.
This seems small but with a microarray with thousands of genes, this
easily produces a bunch of false positives. I looked at 10 chips from
a
real control group arbitrarily labeling 5 chips as control and 5 as
experimental. I would by theory expect 35 false positives and got
exactly 32, that is 32 sitations in which all the low ranks were in
one
group and the high ranks in the other. For a chip with 22000 genes,
you
would expect 175 false positive results by this criteria. Standard
statistical methods would give you a specified type I error rate that
you can count on, it would have found NONE of the genes significant
(i.e. bonferroni adjustment)
This same set of control chips produced 50 false positive results
using
a 2 fold change criteria. Again, these are ALL false positives.
Second,
With respect to t-Tests a couple of people have mentioned "the
problem"
that the t-test denominator may be accidentally "too small" . This is
because the t-test uses an ESTIMATE of the variance from the sample
itself. This is what William Sealey Gossett, otherwise known as
"Student" discovered that prompted him to develop the t-distribution
and
t-test. Gossett or Student was a brewmaster for Guinness breweries in
Dublin and was doing experiments with hops and things and discovered
that the well known "normal distribution" was inaccurate when you
estimated the variance from a sample. He developed the t-distribution
empirically that takes the variability in the variance estimate into
account so that the t-test is ALREADY ADJUSTED to compensate for weird
values in the denominator due to random sampling.
One thing that I think is too often ignored is that different genes
have
different variances, the fact that one gene appears to have a smaller
variance than its neighbors (or a larger one) could be that it
ACTUALLY
DOES have a larger or smaller variance OR it may be due to sampling
variability. The t-test assumes the former but adjusts for the latter
possiblity. It worked then and it works now, it is NOT a problem.
Student's friend, the genius R.A.Fisher took Student's empirical
result
and worked out the theory on which analysis of variance is all based.
This theory has withstood the test of time, it is about 100 years old
and still holds, given the assumptions are correct, t-tests and ANOVA
are still "uniformly most powerful tests".
-.- -.. .---- .--. ..-.
Stephen P. Baker, MScPH, PhD (ABD) (508) 856-2625
Sr. Biostatistician- Information Services
Lecturer in Biostatistics (775) 254-4885 fax
Graduate School of Biomedical Sciences
University of Massachusetts Medical School, Worcester
55 Lake Avenue North
stephen.baker@umassmed.edu
Worcester, MA 01655 USA
------------------------------
.Message: 3
.Date: Mon, 15 Dec 2003 12:11:43 -0000
.From: "michael watson (IAH-C)" <michael.watson@bbsrc.ac.uk>
.Subject: RE: [BioC] ttest or fold change
.To: bioconductor@stat.math.ethz.ch
.Message-ID:
.
<20B7EB075F2D4542AFFAF813E98ACD93028224D1@cl-exsrv1.irad.bbsrc.ac.uk>
.Content-Type: text/plain; charset="utf-8"
.
.
.Why not try the non-parametric t-tests available?
.
.I know all the arguments about a "loss of power" etc, but at the end
of
day, as statisticians and bioinformaticians, .sometimes biologists
come
to us with small numbers of replicates (for very understandable
reasons)
and it is our job to get .some meaning out of that data. Trying to
fit
any kind of statistic involving a p-value to such data is a difficult
and .risky task, and trying to explain those results to the biologist
is
often very difficult.
.
.So here's what happens with the non-parametric tests based on
ranking.
Those genes with the highest |t| are those where .all the replicates
of
one condition are greater than all the replicates of the other
condition. The next highest |t| is .where all but one of the
replicates
of one condition are greater than all the replicates of the other
conddition, etc etc.
.
.OK, so some of these differences could occur by chance, but we're
dealing with often millions of data points and I really .don't think
it's possible to make no mistakes. And curse me if you like, but if i
have a gene expression measurement, .replicated 5 times in two
conditions, and in one condition all five replicates are higher than
the
five replicates of the .other condition, then I believe that that gene
is differentially expressed. And thats easy to find with non-
parametric
t, .and it is easy to explain to a biologist, and at the end of the
day,
is it really wrong to do that?
