top | item 17855171

(no title)

nolemurs | 7 years ago

Repeating a study if, and only if, the study failed to provide the desired result is the most basic p-hacking technique there is. If this isn't p-hacking, I don't know what is!

To give a simple model, suppose you decide an effect is significant if there's only a 5% chance that you'd see this data if the effect didn't exist. If you run the experiment again with the same threshold for significance when don't get the desired result, then the probability of seeing an effect that doesn't exist rises to 9.75% (= 0.05 + 0.95 * 0.05).

The effect isn't merely "not 100% unproblematic" it's a serious problem! You've gone from what looks like a p-value of 5% to a p-value of 9.75%.

The fact that the second study is done with a larger sample is pretty much irrelevant unless it also comes with a higher p-value threshold for you to accept the result.

discuss

order

No comments yet.