[zipdog]

What is more troubling is not that so few of these results results are reproducible, but that it appears almost no-one is trying to reproduce the results of earlier studies. The ability of the scientists who wrote the paper to even get access to the resources necessary to try and reproduce the results is limited.

I'm reminded of Feymann's Cargo Cult Science:

"I was shocked to hear of an experiment done at the big accelerator at the National Accelerator Laboratory, where a person used deuterium. In order to compare his heavy hydrogen results to what might happen with light hydrogen he had to use data from someone else's experiment on light hydrogen, which was done on different apparatus. When asked why, he said it was because he couldn't get time on the program (because there's so little time and it's such expensive apparatus) to do the experiment with light hydrogen on this apparatus because there wouldn't be any new result. And so the men in charge of programs at NAL are so anxious for new results, in order to get more money to keep the thing going for public relations purposes, they are destroying--possibly--the value of the experiments themselves, which is the whole purpose of the thing. It is often hard for the experimenters there to complete their work as their scientific integrity demands."

For anyone who hasn't read it, the whole thing is excellent http://www.lhup.edu/~DSIMANEK/cargocul.htm

[jerf]

If I had to sum up science in one phrase, I wouldn't say anything about the "scientific process" or anything like that. I would say: "Look for reasons why you are wrong, not reasons why you are right." You can always find reasons why you're right. You can always do an astrology forecast and find someone for whom it was dead-on accurate. That's not the problem with astrology, the problem lies in how often it is wrong. It's wrong so often it's useless. But if you only examine the positive evidence in favor of it, you will never come to that conclusion.

The theories that are powerful and worthwhile are the ones that are rarely or never wrong. Can't always get "never". It's a complicated world and we aren't all physicists. But we at least ought to be able to get "rarely", and if you can't, well, I guess it's not science then. That's OK. Unfortunately, not everything is amenable to science, though you can still approach it in this spirit of trying to see how you might be wrong rather than proving yourself right.

Once you start looking around with that standard, it's not hard to see how little science is really being done. Why are we publishing these dubious studies? Because for all the scientific trappings we claim, with statistics and p-values and carefully-written recordings of their putative procedure written in precisely the right way to make it sound like everything was recorded (while still leaving out an arbitrary number of relevant details), we've created a system where we are telling people to look for reasons why they are right... or we won't publish their results. Guess what kind of results we get with that?

If you start from the idea that you need to look for why you are wrong, the scientific method will fall out of that, along with any local adjustments and elaborations you may need, and every discipline, sub-discipline, and indeed at times even individual experiments need adjustments. If you start with "The Scientific Method", but you don't understand where it came from, how to use it, or what it is really telling you, you'll never get true science, just... noise.

[specialist]

"Look for reasons why you are wrong, not reasons why you are right."

I share your worldview. Makes intuitive sense to me. It's intellectually honest. And if I'm wrong about something and no one corrects me, I get kinda grumpy.

Maybe everyone else already knows, but I learned relatively late that it's called Popperian.

http://en.wikipedia.org/wiki/Karl_Popper#Philosophy_of_scien... http://en.wikipedia.org/wiki/Falsifiability

[varjag]

In practical terms, reproducing someone else's work most of the time boils down to redoing someone else's PhD thesis. Which is both not very interesting and doesn't help getting your own research done.

[rdfi]

I agree that reproducing someone else's experiment won't help you get your PhD closer to completion (because you should be doing your own experiments and publishing no matter what, if you expect to land that lecturing position, i.e. publish or perish).

Reproducibility is one of the main principles of the scientific method (those papers shouldn't even been accepted if they don't contain enough information about how to reproduce the experiments they describe)

If an experiment is so complex that it can be compared to redoing someone else's thesis it can be either that the thesis is very simple or the experiment is so complex that it probably proves nothing.

[gwern]

Yes, it's a good piece, but actually the part you should've quoted was the part on the mice experiments...