The Associate Editor of Science who rejected this paper when it was submitted happens to be my wife's friend. They have been having long discussions about this issue. My understanding is that although it's a valid study, it just doesn't bring anything new to the table that is worthy of publication. This paper refutes a 2008 paper that has already been refuted. So the Associate Editor decided there is no reason why this new refutation deserves to be published in Science. That's all.
I don't think there is any case of scientific malpractice here. Keep in mind a high-profile journal like Science is extremely competitive and receives many paper submissions. They have to filter what they choose to publish based on some reason. And it seems to me their reason is perfectly valid.
Now this is turning into a bigger dispute because the researchers started arguing with the Associate Editor on Twitter (I heard the verb "harass".) So he blocked them on Twitter. And this is making them more upset. Etc.
> Keep in mind a high-profile journal like Science is extremely competitive and receives many paper submissions. They have to filter what they choose to publish based on some reason.
This is quite true, but I'd be wary of what you justify based on this fact alone. Discounting all of the interpersonal politics that goes into paper acceptance, the overall editorial policy of "high-profile" journals like this has a considerable effect on the scientific climate.
It would be unfairly cynical (IMHO) to say that the editorial policies will be geared toward whatever the publisher expects to be the most profitable (even in the long term), but on the other hand it would be naïve to expect that they're even trying to do their best for the expansion of human knowledge and the betterment of society.
What I'm getting at is that choosing what to include in Science et al. is a subtle but far-reaching tool, and maybe one better put in the hands of someone at least somewhat impartial, even if it's just some simple popularity contest counting votes on arXiv or similar.
Interesting. The fundamental question is: do journals have an ethical obligation to publish credible results which refute previously published results in the journal?
The authors argue yes. I'm not sure myself -- what is a refutation vs. a different perspective/interpretation or other addition and where do you draw a line?
But it's intriguing to think if this would be a meaningful force for better results -- and could be an incentive to replicate findings you think might not hold up, just from being in a prestigious journal.
I would argue yes. If the answer to a question is sufficient to warrant publication, any significant revision/contradiction to a published result obviously warrants publication.
Further follow-ups need not be published in the same journal, but the first correct/convincing refutation of a published article definitely should be.
> The fundamental question is: do journals have an ethical obligation to publish credible results which refute previously published results in the journal?
The answer is: yes, because that's how science works! And it's especially true if your journal is named "Science".
No. This belongs in a journal dedicated to the field of study, not a general issue journal like Science.
The authors are silly to think otherwise. It is also not surprising that Science published a flashy article without a lot of oomph behind it. It's what they do.
"Science is facing a 'reproducibility crisis' where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests."
> this would seem to be a common occurrence and Science would have to decide whether to devote equal attention to every failed reproduction
I think the answer to this is a rather blatant “of course!”
If Science has the time to support false studies that retrograde the scientific cause, it certainly has the time to publish their corrections and refutations. The scientific process is not about how much stuff you can publish, it's about how much true information you can convey. A journal half the size of Science but twice as trustworthy is at least as valuable, if not vastly more.
> Science would have to decide whether to devote equal attention to every failed reproduction.
Not to every failed reproduction. Only to failures to reproduce results that Science had previously published. And if that means they end up publishing a lot of failures to reproduce, then perhaps they need to re-think their criteria for publishing the original results. After all, why do you think this "reproducibility crisis" exists in the first place?
The original paper was an n=46 study, and they have regression equations there with 5 coefficients. It has been known for a long time that you can't successfully adjust for confounding variables with such small data. The fact it didn't reproduce is easily predicted and hardly noteworthy. Everybody in the field is aware of this issue. This article is just a bunch of academics tooting their own horn. The Journal's response to their request was 'Who cares?', and this is the right answer.
They published it. Clearly everyone didn't know that was nonsense or they knowingly published garbage because it would make for nice press and now they refuse to retract it. Either its shameful behavior in the past, or shameful behavior now, or maybe both.
If my friend asks me what the score in lasts night's game was, and I tell him 4-3, but then later realize that it was actually 4-2, I'll send him a txt to correct the mistake. If your argument is that a lower standard applies to one of the premier scientific journals, I think that's pretty messed up.
> The Journal's response to their request was 'Who cares?', and this is the right answer.
That's what they should have said to the first paper, but since they decided to publish it they can no longer say "who cares" to the correction. They should either retract the original, or publish this.
So what you're saying is Science publishes studies they know are garbage? Good to know, I'll link your comment anytime someone sends me one of their articles.
It sounds like they gave up after Science wouldn't send it out for peer review themselves; whether it's a replication or a refutation, it is not yet even peer reviewed, much less published.
I don't think they provide any compelling reasons why the same journal is obligated to publish replications of any arbitrary study they've published in the past. It wouldn't exactly scale.
The thing I find most interesting about this is that I think people's responses here are being driven in significant part by what was refuted. In particular the bad science was, as is frequently the case, on a political topic and one that I think affirmed the biases of many. In this very thread you can already see people literally arguing that the study that refuted it "was just a data point" which is a direct attempt to try to ignore the far more substantial data provided by the replication effort in showing that the original study was unsupported.
Imagine if the paper that was refuted was one that claimed to show that e.g. humans were not playing as significant a role in global warming as thought. And then Science refused to publish a study that seemed to show conclusively that the original study was, at best, deeply flawed. Would people be responding in the same way?
Given that publishing an article costs money publishing an attempted replication is not likely to happen - if it succeeds it won't happen because we basically already gave you the information, and if it fails it probably won't happen because so what - a replication failed, should they publish it, what if someone comes along and says no we succeeded in our replication should that be published as well because someone else had a replication fail?
How much of the journal should be given over to replications that fail, and then the replications that succeed where another failed, and any potential back and forth?
At any rate a replication study while important for the process of science is not so important for the process of news and as Science is a business and using their resources for publishing replications does not seem likely to increase their profit (but maybe even reduce it) while increasing their cost, I am surprised anyone would even think it should or would happen.
Perhaps what there should be is a reproducibility column, where studies attempting to replicate previously published studies are noted. A paragraph summation could be given. As reproducibility of a finding becomes accepted further replications would be dropped from the column as redundant.
If you don't publish replications and failures to replicate, you aren't publishing science. If a study is worth publishing once, it's worth publishing enough times to establish its result as conclusive. Otherwise, you haven't discovered anything, and the whole scientific process has failed, since the purpose of the scientific process is to discover conclusions.
You're right that in business, there's a disincentive to publish replications, because business treats science publication as literary nonfiction: they're trying to publish the most sensational results, because the most sensational results sell journals. This is one of the strongest arguments that scientific publication shouldn't be a business: making scientific publication a business creates incentives for scientific publications to not publish science.
um, what? It isn’t the obligation of Science to publish any random piece, that the authors conveniently argue is foundational. Null results are useful, but not the most useful thing.
“Best Actor nominee argues why his performance is the most impactful,” etc, etc.
Sure, Science isn't obliged to publish an article which refutes a previous Science article. They could instead simply retract the article which has since been proven wrong.
But to not publish the refutation and not retract the refuted article is to make a deliberate decision to continue to endorse research which has been proven to be incorrect.
A null result is the core mechanism of science. Nothing can be proven true. You can only prove something false, and that is done by providing a counter example. That's what this group did.
Perhaps it’s the political nature of the study but everyone seems to be missing the basic flaw in the argument for publishing this reproduction, that the flaws of the original research ( small sample size, weak correlation, etc ) were published for everyone to see. We’re talking about a science journal here and the general mindset of the comments seems to be equating it with a magazine or something. Educated readers, the real audience of this journal, would’ve read the research ‘against the grain’, if you will, and immediately saw that there was little or nothing there. This begs the question of why publish in the first place, to which an answer might be that it was unique research at the time. The methods pass muster and the data is there to be scrutinized by the reader. These guys are basically taking advantage of the obvious and attempting to, as they say themselves, ‘make their careers’.
But “we” didn’t see the flaws. It has been a high impact study, and people both inside and outside of academia have been working off the assumption that it is true.
What's the modern peer review process for science? Is there an open forum filled with reputed researchers who are willing to write their reviews, ask questions or give suggestions to young researchers who are submitting their work for review in a well-formed format? And is there a role for other bystanders who are observers out of curiosity and may be inspired to become scientific researchers or reviewers themselves? In this age of Internet, isn't this kind of online collaborative open forum going to scale much better?
Why are these age old print journals still in charge of this process with print era processes??
In my opinion, the most reliable science comes from mid-tier journals. The highest impact journals are mixed with the absolute best OR the absolute hyped.
I'm somehow both surprised at the comments here and also not surprised. A remarkably large number of people posting find no problem whatsoever in a supposedly top journal claiming "the field has moved on" from a major and famous study that turns out to be wrong. Heck, I'm not a psychologist and I've heard of this study!
All sorts of important questions are raised by this:
- Should we trust anything published by Science or other top journals at all?
- Is the lackadaisical attitude related to the subject matter? I've read that studies casting conservatives in a good light get replicated almost immediately, and psychology is completely dominated by liberal voters (>90%). Which leads to:
- Why did it take over ten years for such a widely cited study to be replicated? Nobody cared because it fed their own political biases?
And finally:
- What do we do about it? Can academia reform itself at all or is that a lost cause? If so perhaps psychology isn't really worth studying, given the large expense involved and the fact that the outputs so often seem to be noise.
On one hand their research is important and should be published. On the other hand, do they have a right to demand it to be published in the most prestigious journal in the world? High-profile journals reject good articles all the time for various reasons. Especially "the field has moved on" is a grating reason for rejection but is used all the time.
TL;DR: they replicated the famous study about liberals and conservatives reacting differently to threats, finding no difference. Science (a famous journal) refused to publish the null replication.
>We believe that it is bad policy for journals like Science to publish big, bold ideas and then leave it to subfield journals to publish replications showing that those ideas aren’t so accurate after all. Subfield journals are less visible, meaning the message often fails to reach the broader public. They are also less authoritative, meaning the failed replication will have less of an impact on the field if it is not published by Science.
Agreed, but maybe the issue here is journals shouldn't be prestigious. I understand journals' historical usefulness, but the Internet has made them obsolete.
> Agreed, but maybe the issue here is journals shouldn't be prestigious. I understand journals' historical usefulness, but the Internet has made them obsolete.
Prestige and historical usefulness are practically synonyms. You're arguing that prestige shouldn't command respect, but this is like arguing against human nature.
"Science" should recognize its importance to the world at large and take extra care to publish failures of replications of papers that it reports. The more prestigious the journal, the more carefully they ought to follow this proscription, especially if they wish to maintain their prestige.
> TL;DR: they replicated the famous study about liberals and conservatives reacting differently to threats, finding no difference
I think this is a confusing way to word it; it implies that they successfully replicated the original study, and found no difference between their results and the original study. I might say 'they were unable to replicate the famous study about [...]' or 'they attempted to replicated the famous study [...], finding no difference between liberals and conservatives'.
> TL;DR: .... Science (a famous journal) refused to publish the null replication.
They refused to send the null replication out for peer review. I think that is an important difference. The optics make it look more like suppression, rather than a rejection based on quality.
> We should continue to have frank discussions about what we’ve learned over the course of the replication crisis and what we could be doing about it (a conversation that is currently happening on Twitter).
failed to replicate != refutation. the latter is definitive, as in deductive logic, bit science is inductive. Retractions of articles that are no longer convincing in light of new data is the wrong way to go.
I applaud authors for doing a laborious attempt at replication. But, this comes across as sour grapes. They dont make a convincing case for imposing an obligation on Science to peer review their paper.
While I keep my mind open to the idea that science might have new results in this area, there's a certain degree of ideology that goes into coming up with these studies in the first place. If this study had come out at a different point in history, its significance would be less important. However, because of the breaking down of the political order in the United States, people (and especially liberals) are scrambling to explain it. Why are they different from us? How could they believe something different?
While people do have varying anxieties, knowledge, and reactivity to aversive stimuli, there's a stronger prior that political beliefs are based on a few different things.
1) Class Position: People with money or whom believe they are likely to ascend classes trivialize the struggles of others and ascribe problems to individual character rather than systemic design flaws. If an airplane crashes, do we blame the pilots?
2) Culture / Milieu: What ideas are floating around? What do people whom you speak to on a regular basis believe?
3) Personal Experience: Have you had a powerful experience that confirms or contradicts the broader culture?
4) (often but not always most importantly) What will personally benefit you or people you care about?
There's a few other things you could throw in there as well, but Marxists would say to liberals that the reason they are different from you is:
1) They aren't that different from you. You both believe in the integrity of the capitalist system.
2) Often they are from dominant racial or economic castes.
3) There's an intentional campaign on the part of the elites to divide the working class against each other, and one way is via scapegoating minorities and women. Everyone likes to say "No matter how bad it is, at least I'm not X".
4) There is an intentional campaign waged on the part of elites to tie American national identity to its military campaigns overseas.
5) There is an intentional campaign on the part of elites to use race and religion as a wedge issue to divide the people so they do not blame the people causing their problems.
That's not a scientific explanation, but my strong prior is that there's variation amongst people, but we're basically all the same and we're mentally flexible enough to think through adversity and come to our own conclusions.
[+] [-] mrb|6 years ago|reply
I don't think there is any case of scientific malpractice here. Keep in mind a high-profile journal like Science is extremely competitive and receives many paper submissions. They have to filter what they choose to publish based on some reason. And it seems to me their reason is perfectly valid.
Now this is turning into a bigger dispute because the researchers started arguing with the Associate Editor on Twitter (I heard the verb "harass".) So he blocked them on Twitter. And this is making them more upset. Etc.
[+] [-] Y_Y|6 years ago|reply
This is quite true, but I'd be wary of what you justify based on this fact alone. Discounting all of the interpersonal politics that goes into paper acceptance, the overall editorial policy of "high-profile" journals like this has a considerable effect on the scientific climate.
It would be unfairly cynical (IMHO) to say that the editorial policies will be geared toward whatever the publisher expects to be the most profitable (even in the long term), but on the other hand it would be naïve to expect that they're even trying to do their best for the expansion of human knowledge and the betterment of society.
What I'm getting at is that choosing what to include in Science et al. is a subtle but far-reaching tool, and maybe one better put in the hands of someone at least somewhat impartial, even if it's just some simple popularity contest counting votes on arXiv or similar.
[+] [-] blue_devil|6 years ago|reply
Could you cite?
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] crazygringo|6 years ago|reply
The authors argue yes. I'm not sure myself -- what is a refutation vs. a different perspective/interpretation or other addition and where do you draw a line?
But it's intriguing to think if this would be a meaningful force for better results -- and could be an incentive to replicate findings you think might not hold up, just from being in a prestigious journal.
[+] [-] ISL|6 years ago|reply
Further follow-ups need not be published in the same journal, but the first correct/convincing refutation of a published article definitely should be.
[+] [-] CDRdude|6 years ago|reply
[+] [-] amelius|6 years ago|reply
The answer is: yes, because that's how science works! And it's especially true if your journal is named "Science".
[+] [-] Misdicorl|6 years ago|reply
The authors are silly to think otherwise. It is also not surprising that Science published a flashy article without a lot of oomph behind it. It's what they do.
[+] [-] sgustard|6 years ago|reply
https://www.bbc.com/news/science-environment-39054778
As such, this would seem to be a common occurrence and Science would have to decide whether to devote equal attention to every failed reproduction.
[+] [-] Veedrac|6 years ago|reply
I think the answer to this is a rather blatant “of course!”
If Science has the time to support false studies that retrograde the scientific cause, it certainly has the time to publish their corrections and refutations. The scientific process is not about how much stuff you can publish, it's about how much true information you can convey. A journal half the size of Science but twice as trustworthy is at least as valuable, if not vastly more.
[+] [-] pdonis|6 years ago|reply
Not to every failed reproduction. Only to failures to reproduce results that Science had previously published. And if that means they end up publishing a lot of failures to reproduce, then perhaps they need to re-think their criteria for publishing the original results. After all, why do you think this "reproducibility crisis" exists in the first place?
[+] [-] hartator|6 years ago|reply
[+] [-] Gatsky|6 years ago|reply
[+] [-] dagenix|6 years ago|reply
If my friend asks me what the score in lasts night's game was, and I tell him 4-3, but then later realize that it was actually 4-2, I'll send him a txt to correct the mistake. If your argument is that a lower standard applies to one of the premier scientific journals, I think that's pretty messed up.
[+] [-] WillPostForFood|6 years ago|reply
That's what they should have said to the first paper, but since they decided to publish it they can no longer say "who cares" to the correction. They should either retract the original, or publish this.
[+] [-] michaelterryio|6 years ago|reply
[+] [-] sbov|6 years ago|reply
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] tathougies|6 years ago|reply
Science cares, clearly!
[+] [-] wdr1|6 years ago|reply
[+] [-] gota|6 years ago|reply
A better title for laypeople would be "We _refuted_ a Science paper that can't be replicated and Science won't publish it"
[+] [-] scoofy|6 years ago|reply
[+] [-] pera|6 years ago|reply
[+] [-] NoodleIncident|6 years ago|reply
I don't think they provide any compelling reasons why the same journal is obligated to publish replications of any arbitrary study they've published in the past. It wouldn't exactly scale.
[+] [-] rjf72|6 years ago|reply
Imagine if the paper that was refuted was one that claimed to show that e.g. humans were not playing as significant a role in global warming as thought. And then Science refused to publish a study that seemed to show conclusively that the original study was, at best, deeply flawed. Would people be responding in the same way?
[+] [-] bryanrasmussen|6 years ago|reply
How much of the journal should be given over to replications that fail, and then the replications that succeed where another failed, and any potential back and forth?
At any rate a replication study while important for the process of science is not so important for the process of news and as Science is a business and using their resources for publishing replications does not seem likely to increase their profit (but maybe even reduce it) while increasing their cost, I am surprised anyone would even think it should or would happen.
Perhaps what there should be is a reproducibility column, where studies attempting to replicate previously published studies are noted. A paragraph summation could be given. As reproducibility of a finding becomes accepted further replications would be dropped from the column as redundant.
on edit: grammar fix.
[+] [-] kerkeslager|6 years ago|reply
You're right that in business, there's a disincentive to publish replications, because business treats science publication as literary nonfiction: they're trying to publish the most sensational results, because the most sensational results sell journals. This is one of the strongest arguments that scientific publication shouldn't be a business: making scientific publication a business creates incentives for scientific publications to not publish science.
[+] [-] gallerdude|6 years ago|reply
“Best Actor nominee argues why his performance is the most impactful,” etc, etc.
[+] [-] cperciva|6 years ago|reply
But to not publish the refutation and not retract the refuted article is to make a deliberate decision to continue to endorse research which has been proven to be incorrect.
[+] [-] ianferrel|6 years ago|reply
[+] [-] TehCorwiz|6 years ago|reply
[+] [-] perennate|6 years ago|reply
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] scythe|6 years ago|reply
[+] [-] raslah|6 years ago|reply
[+] [-] jtbayly|6 years ago|reply
[+] [-] vinay_ys|6 years ago|reply
Why are these age old print journals still in charge of this process with print era processes??
[+] [-] SubiculumCode|6 years ago|reply
[+] [-] repolfx|6 years ago|reply
All sorts of important questions are raised by this:
- Should we trust anything published by Science or other top journals at all?
- Is the lackadaisical attitude related to the subject matter? I've read that studies casting conservatives in a good light get replicated almost immediately, and psychology is completely dominated by liberal voters (>90%). Which leads to:
- Why did it take over ten years for such a widely cited study to be replicated? Nobody cared because it fed their own political biases?
And finally:
- What do we do about it? Can academia reform itself at all or is that a lost cause? If so perhaps psychology isn't really worth studying, given the large expense involved and the fact that the outputs so often seem to be noise.
[+] [-] bjourne|6 years ago|reply
[+] [-] mLuby|6 years ago|reply
>We believe that it is bad policy for journals like Science to publish big, bold ideas and then leave it to subfield journals to publish replications showing that those ideas aren’t so accurate after all. Subfield journals are less visible, meaning the message often fails to reach the broader public. They are also less authoritative, meaning the failed replication will have less of an impact on the field if it is not published by Science.
Agreed, but maybe the issue here is journals shouldn't be prestigious. I understand journals' historical usefulness, but the Internet has made them obsolete.
[+] [-] tathougies|6 years ago|reply
Prestige and historical usefulness are practically synonyms. You're arguing that prestige shouldn't command respect, but this is like arguing against human nature.
"Science" should recognize its importance to the world at large and take extra care to publish failures of replications of papers that it reports. The more prestigious the journal, the more carefully they ought to follow this proscription, especially if they wish to maintain their prestige.
[+] [-] earenndil|6 years ago|reply
I think this is a confusing way to word it; it implies that they successfully replicated the original study, and found no difference between their results and the original study. I might say 'they were unable to replicate the famous study about [...]' or 'they attempted to replicated the famous study [...], finding no difference between liberals and conservatives'.
[+] [-] CDRdude|6 years ago|reply
They refused to send the null replication out for peer review. I think that is an important difference. The optics make it look more like suppression, rather than a rejection based on quality.
[+] [-] jxramos|6 years ago|reply
What's this twitter conversation they reference?
[+] [-] your-nanny|6 years ago|reply
[+] [-] your-nanny|6 years ago|reply
[+] [-] tehjoker|6 years ago|reply
While people do have varying anxieties, knowledge, and reactivity to aversive stimuli, there's a stronger prior that political beliefs are based on a few different things.
1) Class Position: People with money or whom believe they are likely to ascend classes trivialize the struggles of others and ascribe problems to individual character rather than systemic design flaws. If an airplane crashes, do we blame the pilots?
2) Culture / Milieu: What ideas are floating around? What do people whom you speak to on a regular basis believe?
3) Personal Experience: Have you had a powerful experience that confirms or contradicts the broader culture?
4) (often but not always most importantly) What will personally benefit you or people you care about?
There's a few other things you could throw in there as well, but Marxists would say to liberals that the reason they are different from you is:
1) They aren't that different from you. You both believe in the integrity of the capitalist system.
2) Often they are from dominant racial or economic castes.
3) There's an intentional campaign on the part of the elites to divide the working class against each other, and one way is via scapegoating minorities and women. Everyone likes to say "No matter how bad it is, at least I'm not X".
4) There is an intentional campaign waged on the part of elites to tie American national identity to its military campaigns overseas.
5) There is an intentional campaign on the part of elites to use race and religion as a wedge issue to divide the people so they do not blame the people causing their problems.
That's not a scientific explanation, but my strong prior is that there's variation amongst people, but we're basically all the same and we're mentally flexible enough to think through adversity and come to our own conclusions.
[+] [-] unknown|6 years ago|reply
[deleted]