WingNews logo WingNews
top | item 845216

(no title)

efaith | 16 years ago

"If a study, significant at p=0.05, is replicated again at p=0.05, then the odds of that happening when there actually is no effect (assuming independence) are the product of the two (0.0025)."

I think that must be approximately right, but I'm not sure it is precisely right. Imagine you do a very large number of studies and suppose the null hypothesis is in fact true. P will always be less than 1, maybe slightly less than one - or suppose that it is 1 occasionally. This is because P varies from 0 to 1. Now go ahead and multiply these Ps, and if you've done enough studies, the product is going to be very small even if all the individual Ps are close to 1 and even though the null hypothesis is true.

So I don't think a simple multiplication is exactly the right formula, though it does seem as though it must be about right.

I googled the question and found a formula for combining P of two independent studies, but it's not simple multiplication. You start with the Ps, find the corresponding Zs (I do not know what those are), then add them and divide by sqrt(2). This is your new Z, and then you take the corresponding P. Also, it requires that the Ps must be one-tailed, so it is not a fully general formula. I do not understand the Zs but my point is that if it were simply a matter of multiplying the Ps then why go to the trouble of adding Zs? I found it here:

http://books.google.com/books?id=nxOFMQYMIlgC&lpg=PA527&...

As for why it doesn't match exactly the familiar formula for combining independent probabilities (i.e. you simply multiply them), I think the answer lies in the nature of P. P is not really "the probability of that result" but "the probability of a result that is at least that extreme", and this subtly different meaning results in a different way in which the individual Ps must be combined.

discuss

order

idm|16 years ago

You're right about my claim being approximate - the biggest problem with what I said was "assuming independence." If you're replicating a study, it's definitely not going to be a completely independent event. Usually, you copy at least part of the methods, and that right there is a major dependency.

As for the definition of P, that's something they address in the original article, and you are in agreement with the authors: P is _definitely_ not "the probability of that result." In its interpretation, P is the chance that you accept the null hypothesis when it is false (i.e. there is no effect, but your data randomly showed an effect).