You can also see that as an opportunity, at least for those who stay in academia: all fundamentals are up for reevaluation, and replication studies haven't been popular until recently, so there's bound to be some high-impact, low-hanging fruit.
It's worse than that. Imagine being in a situation where if you assumed everything you learned was incorrect you would be "more right" than if you assumed it was correct.
people are self-interested not truth interested. most scientific fields are initiated by a small handful of geniuses, but over time become a type of cloistered bureaucracy. your standard university education teaches you there are two paths - you make significant contributions and are remembered long after you die, or you plug in as a fresh node into the existing structure and are rewarded with a comfortable life, just check your ethics at the door.
this is largely due to the failure of idealism as a philosophical framework to set up any type of enduring political, economic or secular social structure. the blame lies on hegel or perhaps marx's interpretation of hegel getting stuck in praxis quagmires. which is why the turn towards asia and the new age movement which emerges from it in the early 1900s is essentially anti-science. the spiritual successor of communism is the idea of fusing the east and west traditions to form the new enlightened human, who sees no race, religion or gender but simply behaves as the universal light and creator. it has largely been a failure, as folding china into the world community has been against the pragmatic self-interest of various sensitive american industries - weapons, space, communication, high finance, which can exert sufficient power on the american leadership structures to avoid any type of idealism about world peace. there are no more enemies or allies just competitive co-morbidity, actors fraying the ropes they are tugging on until something breaks, like the ussr in the 80s.
it's only a problem if you think humans are equal. if you shift the view that humans are unequal and life is deeply unfair, then it's normal to see the next 100 years as a struggle for dominance between superstition and science, between a world led by north america or eurasia, between the new dominoes of fascist nationalism and disinterested international capitalism. it's a hot peace, which will end up with one temporary victor, possibly colonizing mars - or just landing there and coming back, before a new power emerges to challenge the old empire. i would argue that this is a bi-product of the 'correct' interpretation of hegel, the one that isn't taught directly in schools, the master and slave dialectic.
> Combined with industry-wide pressures to publish, the replication crisis was inevitable.
> The replication crisis, if nothing else, has shown that productivity is not intrinsically valuable.
I think this is important to focus on - the point of universities has become to produce profit, and to give people degrees that are profitable, and to appear to be able to do those things. This has very little to do with producing research with verifiable results. It's much more to do with getting students into the funnel by making people with tenure appear as productive as possible.
AFAICT this is not about profit in a commercial sense, as in selling goods.
It's more like overfitting the target function of publishing impactful research. A bit of p-hacking, a bit of cutting corners in experimental setup, a sloppy null hypothesis check, and you honestly believe you see an effect! Everyone is happy: you, your adviser, lab's administration, the journal where you publish the paper.
But if you carefully check for everything, then find no effect, you kill an interesting hypothesis, your paper is hard to publish, "you are not making progress", and nobody is happy.
The article argues the replication crisis is somehow unique to psychology, but it's not.
As the Scott Alexander essay makes clear, it also affects psychiatry and it's apparently the case that many areas of medicine have this problem. Biotech studies are also hives of replication failures.
Even AI research has had replication failures and that's based on running theoretically deterministic software on theoretically deterministic machines!
Some of this is that accurately describing studies and replicating them is hard. But some of it is that academics aren't incentivised to find the truth, but rather, to make it appear that academics know a lot of things.
I'm unsurprised by the replication crisis. I've seen scientists p-hacking, and even choosing inappropriate statistical tests because it gave a "better" result. Example: using a non-parametric "U"-test when a parametric "T"-test gave a non-significant result. Along with low numbers of replicates. Statistical analysis is only useful and meaningful if your data is of good quality, and you use a statistical test appropriate for the data. If your data is of marginal quality, then I'm afraid that it's simply unworthy of publication. But when your career hangs in the balance, stuff like this gets through.
The major problem with current scientific practice is that good practice is actively penalised. I used to work in industry, and I was shocked at the lax standard of work, particularly with respect to accuracy and precision, of wet lab scientists in university research settings. I mentioned this to a few postdocs over the years and paraphrased was told that "if it's good enough to publish, then it's good enough for me", which if you think about it, is actually quite a low bar. Most of the people were fully aware they were doing sloppy work, but didn't care.
How can it be improved? I think there are two sides to this coin. Firstly, good practice has to be encouraged and rewarded, and sloppiness penalised. That requires a culture change in the laboratories. Too many PIs don't care about what happens in their labs so long as "good" results are being generated by their underlings. They don't look after instrument calibration and ensure that people are working to GLP standards. In industrial labs, we had to send samples off to reference labs, analyse random samples provided to us, and undergo inspections and audits to prove we were providing correct analyses. Maybe academic labs should be obligated to prove themselves as well, or lose their funding?
Secondly, a project delivering negative results should not be a career-ending move. Failing experiments does not necessarily mean one is a bad scientist. But right now, the incentives are to spin all results in a positive light, even if it means publishing bad science, because that's what it takes to keep the funding coming in. Success should be rewarded, but I think our criteria for what success is need to be recalibrated to reduce charlatans abusing the system for their own benefit. Publishing a paper isn't enough; it's got to be replicable independently.
The point about not having models is a very important point. Psychology is so sure that the mind exists and has certain structural properties, but to what extent is it empirical?
A model is useful if its behavior matches observed behavior reasonably well. So long as the abstraction doesn't leak, it doesn't need to represent the mechanisms correctly. Even if a "mind" doesn't exist, our mind-model may still be useful.
Regarding Bem's precognition experiments, an extract from Wikipedia:
In a 2017 follow-up article in Slate magazine on the "Feeling the Future" experiments, Bem is quoted as saying, “[...] If you looked at all my past experiments, they were always rhetorical devices. I gathered data to show how my point would be made. I used data as a point of persuasion, and I never really worried about, ‘Will this replicate or will this not?’”"[42] While fellow psychologist Stuart Vyse sees this statement as coming "remarkably close to an outright admission of p-hacking", he also notes that Bem "has been given substantial credit for stimulating the movement to tighten the standards for research" such as that taking place in open science.[43]
"There is some evidence, however, for the hypothesis that people can feel the future with emotionally valenced nonerotic stimuli, with a Bayes factor of about 40. Although this value is certainly noteworthy, we believe it is orders of magnitude lower than what is required to overcome appropriate skepticism of ESP."
–
https://link.springer.com/article/10.3758%2Fs13423-011-0088-...
A low threshold for statistical significance won't solve all problems of p-Hacking, and the implementation of Bayesian methods doesn't seem to promising as well. Will be interesting to see how the field of psychology is going to change over time.
I find this article quite hard to read (poor writing, grammatical errors making sentences incoherent), but the key thesis seems to be that because psychology has embraced empiricism as its philosophy of science, and not required any theory to explain and interpret empirical results, it is uniquely susceptible to replication failure. In a sense, the replication crisis in psychology is string evidence that empiricism is an incorrect theory of science: if it were, psychology would be doing great—just as well as theory-laden sciences like physics, chemistry and biology. Instead, what we see is that the more a science takes coherent, broad theories seriously, the better if fares as a scientific endeavor. This matches a Popperian/Deutchian philosophy of science. The most interesting thing about the crisis is that it is, effectively, a scientific experiment about the nature of science.
[+] [-] mensetmanusman|6 years ago|reply
Imagine finishing graduate school and then suddenly learning that over half of what you learned is probably not true... but not knowing which half
[+] [-] turbinerneiter|6 years ago|reply
[+] [-] Ao7bei3s|6 years ago|reply
[+] [-] tengbretson|6 years ago|reply
[+] [-] x220|6 years ago|reply
(please excuse the shitpost effects, the video is still useful)
[+] [-] cerealbad|6 years ago|reply
this is largely due to the failure of idealism as a philosophical framework to set up any type of enduring political, economic or secular social structure. the blame lies on hegel or perhaps marx's interpretation of hegel getting stuck in praxis quagmires. which is why the turn towards asia and the new age movement which emerges from it in the early 1900s is essentially anti-science. the spiritual successor of communism is the idea of fusing the east and west traditions to form the new enlightened human, who sees no race, religion or gender but simply behaves as the universal light and creator. it has largely been a failure, as folding china into the world community has been against the pragmatic self-interest of various sensitive american industries - weapons, space, communication, high finance, which can exert sufficient power on the american leadership structures to avoid any type of idealism about world peace. there are no more enemies or allies just competitive co-morbidity, actors fraying the ropes they are tugging on until something breaks, like the ussr in the 80s.
it's only a problem if you think humans are equal. if you shift the view that humans are unequal and life is deeply unfair, then it's normal to see the next 100 years as a struggle for dominance between superstition and science, between a world led by north america or eurasia, between the new dominoes of fascist nationalism and disinterested international capitalism. it's a hot peace, which will end up with one temporary victor, possibly colonizing mars - or just landing there and coming back, before a new power emerges to challenge the old empire. i would argue that this is a bi-product of the 'correct' interpretation of hegel, the one that isn't taught directly in schools, the master and slave dialectic.
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] tomlock|6 years ago|reply
> The replication crisis, if nothing else, has shown that productivity is not intrinsically valuable.
I think this is important to focus on - the point of universities has become to produce profit, and to give people degrees that are profitable, and to appear to be able to do those things. This has very little to do with producing research with verifiable results. It's much more to do with getting students into the funnel by making people with tenure appear as productive as possible.
[+] [-] nine_k|6 years ago|reply
It's more like overfitting the target function of publishing impactful research. A bit of p-hacking, a bit of cutting corners in experimental setup, a sloppy null hypothesis check, and you honestly believe you see an effect! Everyone is happy: you, your adviser, lab's administration, the journal where you publish the paper.
But if you carefully check for everything, then find no effect, you kill an interesting hypothesis, your paper is hard to publish, "you are not making progress", and nobody is happy.
Crooked incentives, crooked results :(
[+] [-] repolfx|6 years ago|reply
As the Scott Alexander essay makes clear, it also affects psychiatry and it's apparently the case that many areas of medicine have this problem. Biotech studies are also hives of replication failures.
Even AI research has had replication failures and that's based on running theoretically deterministic software on theoretically deterministic machines!
Some of this is that accurately describing studies and replicating them is hard. But some of it is that academics aren't incentivised to find the truth, but rather, to make it appear that academics know a lot of things.
[+] [-] rleigh|6 years ago|reply
The major problem with current scientific practice is that good practice is actively penalised. I used to work in industry, and I was shocked at the lax standard of work, particularly with respect to accuracy and precision, of wet lab scientists in university research settings. I mentioned this to a few postdocs over the years and paraphrased was told that "if it's good enough to publish, then it's good enough for me", which if you think about it, is actually quite a low bar. Most of the people were fully aware they were doing sloppy work, but didn't care.
How can it be improved? I think there are two sides to this coin. Firstly, good practice has to be encouraged and rewarded, and sloppiness penalised. That requires a culture change in the laboratories. Too many PIs don't care about what happens in their labs so long as "good" results are being generated by their underlings. They don't look after instrument calibration and ensure that people are working to GLP standards. In industrial labs, we had to send samples off to reference labs, analyse random samples provided to us, and undergo inspections and audits to prove we were providing correct analyses. Maybe academic labs should be obligated to prove themselves as well, or lose their funding?
Secondly, a project delivering negative results should not be a career-ending move. Failing experiments does not necessarily mean one is a bad scientist. But right now, the incentives are to spin all results in a positive light, even if it means publishing bad science, because that's what it takes to keep the funding coming in. Success should be rewarded, but I think our criteria for what success is need to be recalibrated to reduce charlatans abusing the system for their own benefit. Publishing a paper isn't enough; it's got to be replicable independently.
[+] [-] lidHanteyk|6 years ago|reply
[+] [-] xapata|6 years ago|reply
[+] [-] x220|6 years ago|reply
[+] [-] interblue|6 years ago|reply
In a 2017 follow-up article in Slate magazine on the "Feeling the Future" experiments, Bem is quoted as saying, “[...] If you looked at all my past experiments, they were always rhetorical devices. I gathered data to show how my point would be made. I used data as a point of persuasion, and I never really worried about, ‘Will this replicate or will this not?’”"[42] While fellow psychologist Stuart Vyse sees this statement as coming "remarkably close to an outright admission of p-hacking", he also notes that Bem "has been given substantial credit for stimulating the movement to tighten the standards for research" such as that taking place in open science.[43]
– [42]: https://redux.slate.com/cover-stories/2017/05/daryl-bem-prov... 43: https://web.archive.org/web/20180805142806/https://www.csico...
Also from one of the articles' sources:
"There is some evidence, however, for the hypothesis that people can feel the future with emotionally valenced nonerotic stimuli, with a Bayes factor of about 40. Although this value is certainly noteworthy, we believe it is orders of magnitude lower than what is required to overcome appropriate skepticism of ESP." – https://link.springer.com/article/10.3758%2Fs13423-011-0088-...
A low threshold for statistical significance won't solve all problems of p-Hacking, and the implementation of Bayesian methods doesn't seem to promising as well. Will be interesting to see how the field of psychology is going to change over time.
[+] [-] StefanKarpinski|6 years ago|reply