If you want to know whether a drug is more effective than placebo, the answer to that question depends on both the data collected in a study and the initial study design. There’s a reason why it’s meaningless to say “that was unlikely” after somebody says they were born on January 1, or after getting a two-factor code that is the same number six times. There’s nothing special about those particular events except for the fact that we noticed them. Since we live in a single instance of the universe where they have already happened, they have probability 1. At the same time, on any given instance they have probability 1/365ish or 1/10000. The difference between these two interpretations of the probability is the same difference as having a good experimental design vs a flawed experimental design where you repeat the experiment until you get the results you want to see.
pdonis|2 years ago
But the Bayesian point is that, if you use Bayesian statistics, this doesn't work. Except by outright lying about their experimental protocol or the data that was actually collected (for example, only reporting the successful trial at the end and not all the failed ones the preceded it), an experimenter cannot "fool" you into accepting a hypothesis not justified by the data. They can point to the one successful trial all they want, and make up stories about how the previous failed trials were somehow different, and the Bayesian simply does not care. The Bayesian just looks at the entire corpus of data and finds that it doesn't support the hypothesis, and that's it.