Lesson - just because people claim they are doing "science" doesn't mean they are doing either good or correct science. Subtle unchallenged assumptions, cause-and-effect confusion, inadequate root-cause analysis, invalid inferential logic and questionable proof-by-statistics.
Maybe a better question - should psychology even be considered scientific or a science in the first place?
Killing the messenger, really. This stuff has been documented in all sorts of fields, including pharmacology, neuroscience, oncology, you name it. I think there's been posts here on HN lately about replicability problems with AI research?
Psychology is fuzzy because of the subject matter, but (appropriately I think) it's also better at turning the microscope on itself (meta-analysis really has its origins in psychology).
Pretending this doesn't happen elsewhere is dangerous. Maybe the rates vary from field to field, but psychology isn't alone. If you applied that standard to every field there would be almost nothing left except maybe physics and some other closely related fields.
Sloppy science is done in every field, natural sciences included. The article even leads with the statement that psychology experience more reversals due to being exceptionally open in terms of sharing code and data compared to other social sciences.
Reversals through replication failure is scientific progress. Your “better” question is nothing of the sort, just lazy contrarianism.
>>>> Maybe a better question - should psychology even be considered scientific or a science in the first place?
Psychology has embraced scientific methodology. However, it has struggled to produce a reliable scientific knowledge base. There are some successes in psychology, that tempt one to apply the label "scientific," but they are in areas that might be controversial, such as advertising and manipulation.
Everything you say is true, probably for all fields. There is undeserved faith in peer review processes (https://www.vox.com/2015/12/7/9865086/peer-review-science-pr...), inadequate understanding of statistics (or maybe purposefully poor application of statistics), poor incentive structures, and other issues that corrupt "science". This is why I always cringe a little when someone says "trust the science".
Even if people have impressive titles at institutions and landmark discoveries to their name, doesn't mean they are doing either good or correct science.
I notice that the page says that Daryl Bem's experiements on precognition have no good evidence. I poked around a bit and immediately found this paper on a metanalysis from 2016 on the topic.
To encourage replications, all materials needed to conduct them were made available on request. We here report a meta-analysis of 90 experiments from 33 laboratories in 14 countries which yielded an overall effect greater than 6 sigma, z = 6.40, p = 1.2 × 10 -10 with an effect size (Hedges’ g) of 0.09. A Bayesian analysis yielded a Bayes Factor of 5.1 × 10 9, greatly exceeding the criterion value of 100 for “decisive evidence” in support of the experimental hypothesis. When DJB’s original experiments are excluded, the combined effect size for replications by independent investigators is 0.06, z = 4.16, p = 1.1 × 10 -5, and the BF value is 3,853, again exceeding the criterion for “decisive evidence.” The number of potentially unretrieved experiments required to reduce the overall effect size of the complete database to a trivial value of 0.01 is 544, and seven of eight additional statistical tests support the conclusion that the database is not significantly compromised by either selection bias or by intense “ p-hacking”—the selective suppression of findings or analyses that failed to yield statistical significance. P-curve analysis, a recently introduced statistical technique, estimates the true effect size of the experiments to be 0.20 for the complete database and 0.24 for the independent replications, virtually identical to the effect size of DJB’s original experiments (0.22) and the closely related “presentiment” experiments (0.21). We discuss the controversial status of precognition and other anomalous effects collectively known as psi.
This is completely unusable to me. The title says reversals yet the body doesn’t seem to contain reversals…
“No strong evidence of X” is a lot different than “Strong evidence against X.” Any study of anything (true or not) will conclude “insufficient evidence” if the sample size is too small, or there is no control group.
I think it’s the general consensus that anything in research, especially psychology is undecided until what’s called a “meta-study” is released.
Also, Perhaps universities should have unaffiliated professors do each other’s experiments and analyze their data (all videotaped) to avoid the inherent conflict of interest in research.
Multiple replications are no guarantee of strong evidence. The first example (elderly priming) in the article shows that. It's the old file drawer problem.
Interesting. I have a Master's in Cognitive Psychology (from long ago, the 1990s), but it was in the domain Human Factors, so I view "cognitive psychology" in a very different light than these example. See some common examples like:
Also things involving haptics, echo-location, alarm response, etc...
What these things had in common, I think, is they treated the human mind as pretty much a black box, the dependent measures were not much subject to bias, and the experiments were easy to run on large samples of people. Such findings are pretty useful when designing things like nuclear power plants, airplane cockpits, shock trauma resuscitation bays, etc...
The replication crisis in psychology has been on for a solid decade now. We need to stop treating poor statistical practice as negligence and start treating it as fraud.
Just add a statistician to review every single paper in every field for statistical error before publication. Every journal that doesn't do this should be deemed unscientific and lose all its status.
Of course this wont happen since many "scientists" got to their current position by publishing non-results using bad statistical methods.
And no, this wouldn't be too expensive at all. Is it really too expensive to check that our record of human knowledge is correct? It wouldn't cost more than a couple of hundred dollars per paper, basically nothing compared to writing the paper.
"No good evidence for multiple intelligences (in the sense of statistically independent components of cognition)."
But shouldn't the burden of proof be the other way around? Why is it more reasonable to believe that ability in different cognitive activities can be abstracted into a single component called "intelligence" than it is to believe that performance in unrelated cognitive tasks would be unrelated?
> Why is it more reasonable to believe that ability in different cognitive activities can be abstracted into a single component called "intelligence" than it is to believe that performance in unrelated cognitive tasks would be unrelated?
Presumably because both hypotheses can be tested and one of them came out ahead?
Any science you hear about on the news is by selection bias more likely to be wrong than the science you never hear about. In short: news is destroying our trust in science by picking the worst examples of it for clicks.
[+] [-] tus89|4 years ago|reply
Maybe a better question - should psychology even be considered scientific or a science in the first place?
[+] [-] derbOac|4 years ago|reply
Psychology is fuzzy because of the subject matter, but (appropriately I think) it's also better at turning the microscope on itself (meta-analysis really has its origins in psychology).
Pretending this doesn't happen elsewhere is dangerous. Maybe the rates vary from field to field, but psychology isn't alone. If you applied that standard to every field there would be almost nothing left except maybe physics and some other closely related fields.
[+] [-] undreren|4 years ago|reply
Reversals through replication failure is scientific progress. Your “better” question is nothing of the sort, just lazy contrarianism.
[+] [-] analog31|4 years ago|reply
Psychology has embraced scientific methodology. However, it has struggled to produce a reliable scientific knowledge base. There are some successes in psychology, that tempt one to apply the label "scientific," but they are in areas that might be controversial, such as advertising and manipulation.
[+] [-] throwawaysea|4 years ago|reply
Yet at the same time, I think psychology is especially susceptible to these problems, because it is a more fuzzy "social science" (more like sociology, less like physics). It suffers immense bias due to those who select themselves into the field and also how they study it. Psychology researchers typically conduct non-generalizable experiments with immense sampling bias (https://www.psychologytoday.com/us/blog/non-weird-science/20...). Then there are replication problems (https://www.theatlantic.com/science/archive/2018/11/psycholo...). They're also not making any progress in addressing these issues because of meta cultural problems about acknowledging problems in the field (https://www.wired.com/2016/03/psychology-crisis-whether-cris...).
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] BurningFrog|4 years ago|reply
You can do good scientific research in that field. But it's much harder than in other fields.
[+] [-] BurningFrog|4 years ago|reply
[+] [-] vajrabum|4 years ago|reply
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4706048/
Any thoughts on this? This paper says:
To encourage replications, all materials needed to conduct them were made available on request. We here report a meta-analysis of 90 experiments from 33 laboratories in 14 countries which yielded an overall effect greater than 6 sigma, z = 6.40, p = 1.2 × 10 -10 with an effect size (Hedges’ g) of 0.09. A Bayesian analysis yielded a Bayes Factor of 5.1 × 10 9, greatly exceeding the criterion value of 100 for “decisive evidence” in support of the experimental hypothesis. When DJB’s original experiments are excluded, the combined effect size for replications by independent investigators is 0.06, z = 4.16, p = 1.1 × 10 -5, and the BF value is 3,853, again exceeding the criterion for “decisive evidence.” The number of potentially unretrieved experiments required to reduce the overall effect size of the complete database to a trivial value of 0.01 is 544, and seven of eight additional statistical tests support the conclusion that the database is not significantly compromised by either selection bias or by intense “ p-hacking”—the selective suppression of findings or analyses that failed to yield statistical significance. P-curve analysis, a recently introduced statistical technique, estimates the true effect size of the experiments to be 0.20 for the complete database and 0.24 for the independent replications, virtually identical to the effect size of DJB’s original experiments (0.22) and the closely related “presentiment” experiments (0.21). We discuss the controversial status of precognition and other anomalous effects collectively known as psi.
[+] [-] zug_zug|4 years ago|reply
“No strong evidence of X” is a lot different than “Strong evidence against X.” Any study of anything (true or not) will conclude “insufficient evidence” if the sample size is too small, or there is no control group.
I think it’s the general consensus that anything in research, especially psychology is undecided until what’s called a “meta-study” is released.
Also, Perhaps universities should have unaffiliated professors do each other’s experiments and analyze their data (all videotaped) to avoid the inherent conflict of interest in research.
[+] [-] ghostbrainalpha|4 years ago|reply
https://www.ox.ac.uk/news/2019-01-15-technology-use-explains...
[+] [-] JadeNB|4 years ago|reply
That shouldn't be taken as evidence of the opposite, though (not that you said that!).
[+] [-] stkdump|4 years ago|reply
[+] [-] exporectomy|4 years ago|reply
[+] [-] derbOac|4 years ago|reply
[+] [-] newsbinator|4 years ago|reply
[+] [-] rickspencer3|4 years ago|reply
https://en.wikipedia.org/wiki/Stroop_effect https://en.wikipedia.org/wiki/Fitts%27s_law
Also things involving haptics, echo-location, alarm response, etc...
What these things had in common, I think, is they treated the human mind as pretty much a black box, the dependent measures were not much subject to bias, and the experiments were easy to run on large samples of people. Such findings are pretty useful when designing things like nuclear power plants, airplane cockpits, shock trauma resuscitation bays, etc...
[+] [-] NoImmatureAdHom|4 years ago|reply
[+] [-] username90|4 years ago|reply
Of course this wont happen since many "scientists" got to their current position by publishing non-results using bad statistical methods.
And no, this wouldn't be too expensive at all. Is it really too expensive to check that our record of human knowledge is correct? It wouldn't cost more than a couple of hundred dollars per paper, basically nothing compared to writing the paper.
[+] [-] amelius|4 years ago|reply
[+] [-] kar5pt|4 years ago|reply
"No good evidence for multiple intelligences (in the sense of statistically independent components of cognition)."
But shouldn't the burden of proof be the other way around? Why is it more reasonable to believe that ability in different cognitive activities can be abstracted into a single component called "intelligence" than it is to believe that performance in unrelated cognitive tasks would be unrelated?
[+] [-] jhgb|4 years ago|reply
Presumably because both hypotheses can be tested and one of them came out ahead?
[+] [-] exporectomy|4 years ago|reply
Gardner's multiple intelligences are just made up. They didn't appear out of some analysis. He just imagined them and declared them to exist.
[+] [-] machinehermitt|4 years ago|reply
It is not a scientific category of inquiry by itself though the way a theory of nodes does make sense outside of graph theory.
[+] [-] arsome|4 years ago|reply
I should probably stop listening to those.
[+] [-] c3600608-467b|4 years ago|reply
Any science you hear about on the news is by selection bias more likely to be wrong than the science you never hear about. In short: news is destroying our trust in science by picking the worst examples of it for clicks.