As a reminder, peer review is not designed adversarially. It is not supposed to catch people who are fraudulent before they publish---that's the point of replication (which happens rarily). Peer review is designed to ensure a the publication as a packaged work of science describes a valid experiment. But you're necessarily assuming that the authors did what they said they did. As a peer reviewer, you're ensuring that what the authors said they did constitutes valid science.
In practice, peer review isn't even that. Most referees are not double-checking your statistical analysis. What they focus on is whether the research is interesting and has it appropriately considered relevant literature. Even that is not always done carefully.
I don't think peer review is about ensuring anything. It's about spending some time to improve the quality of the paper to save time for everyone else. It's more about providing editorial services than about the reported work itself.
A research paper is supposed to be a honest report of best efforts to study a topic. If it's not, that's a problem that can't be solved by having a few people spending a few hours with the report. The paper is not the final word on anything anyway. If you read a paper expecting to learn something about the world, you are doing something wrong.
As a reviewer, you determine whether the paper is interesting and relevant to the venue. You report any issues you spot that should be corrected and any things you believe that could be improved. And if you get any ideas you feel like sharing, you may share them as well. And that's it.
> As a reminder, peer review is not designed adversarially.
I beg to differ. Peer review is an adversarial process. The author(s) of the paper are making statements and report findings proving them by logic or/and data and analysis. The priors while reviewing a science paper is "wrong" until proven "correct". This also covers accuracy and veracity of the data and analysis. The only thing a peer review is not is assigning intent or blame. It is not the job of a reviewer to look for fraud when simple incompetence could explain it. But after reading this article I will add fraud dimension to a list of fallacies I am looking for while peer-reviewing a manuscript.
Disagree. From my experience as a scientist there isn't a manual that says what peer review is and isn't supposed to be. Of course, some journals have such guidelines (not that all reviewers read them), and the editors who assign the reviews are free to ignore a rogue reviewer. But there is a set of explicit or tacit standards of good science, and any of it is fair game in peer review:
1. Validity of the experiment
2. Interestingness / novelty of the work
3. Appropriate choice of methods and correctness / believability of the results
4. Signs of outright fraud (suspicious figures, etc.)
Absolutely if I notice some weird Photoshop artifacts in a figure, or some other obvious sign of fake data, I'm going to call that out. I'm probably an outlier on this next one, but I've even been tempted to reject articles just for having egregiously, unreadably bad writing. I know this will be regarded as bias against people whose first language is not English, but if the writing is so bad that it stands in the way of making sense of the article, and the authors can't be bothered to get a decent editor, it's not a worthy contribution to the (English language) scientific literature.
> "Going by these numbers, roughly one in 1,000 papers gets retracted [..] that something more like one in 50 papers has results which are unreliable because of fabrication, plagiarism or serious errors."
I'd say these are underestimates. Let me add that 80%+ of papers are useless. The only "value" they provide is to the person getting academically promoted and/or building their publishing portfolio/cred.
As far as I can see this is mostly an incentives problem. Bureaucratic control of academic hiring has ended up emphasizing the short term measurable (e.g. #of pages published, so-called impact factors, etc.) over the long term, with pretty predictable results.
Medical research in particular is fraught with another set of problems; the default clinical pathway gives a weak at best grounding in science, and even the MD/PhD programs have been gamed to some degree. There are definite counterexamples (lots!) but there are also a lot of clinicians with incentive to produce research but little skill in it and even less time available...
Some of that 80% of papers are also valuable to politicians and bureaucrats pushing biased narratives on the public. No matter what position they want to take they can cherry pick some low-quality research to justify it with a veneer of "science".
How are we defining “useless” here? Papers that are wrong and fraudulent? Or is it broader and any paper we currently can’t do anything with or yields a negative result?
If you were to ask experts in a given subfield which papers are reliable, I'm sure they would be able to tell you. The problem is that there's no process in science for expert consensus to make it to out to doctors/laypeople.
People assume that peer review means a paper is good, which couldn't be farther from the truth. Science journalists aren't any better, they care more about hype than consensus. Honestly, it's dangerous to give a random peer reviewed article to someone who doesn't have broad knowledge of the field.
Maybe we need middle-ground journals that publish review articles at the level of a Scientific American reader?
> People assume that peer review means a paper is good ...
"Conclusions: Parachute use did not reduce death or major traumatic injury when jumping from aircraft in the first randomized evaluation of this intervention."
It’s not a given experts know what papers are reliable. Here’s a paper from Genentech https://pubmed.ncbi.nlm.nih.gov/15385631/ with 500 citations which my advisors swore is reliable because “they trust the authors.” Don’t mind the fact that the central thesis of the article is barely supported by the last figure in which they conveniently average data from completely different sets of experiments for each time point.
And did I mention my advisors released a drug into the market just recently? Lol.
That's one of the intended purposes of review journals like Annual Reviews (https://www.annualreviews.org/). There are some pretty big practical issues with them though:
1. Sometimes the latest article on a subject was written 2/5/10 years ago. Depending on quickly the field is moving, that review could be perfectly acceptable, completely obsolete, or anywhere in between.
2. Sometimes the author themselves is just out of date with the field. It's difficult to identify this without deep prior familiarity.
3. Paywalls.
I don't know what the solutions are beyond "making experts accessible for questions" and other public outreach things.
I maintain that the best analogy for life sciences research is naval exploration. Some people tell tales of their great voyages but never set foot on a boat. Some sailed, yet found nothing, and still spin a story to avoid embarrassment. And some actually found America. Science is obsessed with these stories enshrined in journal articles, which are elevated to the status of artefacts of knowledge, complete with Excel spreadsheets in the supplementary info. Creating such artefacts determines everything in your scientific career. That is the actual problem in my opinion. Of course people will want to fake these powerful status symbols.
I recall reading -somewhere - about a similar problem in psychology journals. The problem there is worse, in terms of correctness, because the journals don’t publish negative results, including when the negative results disprove a previously published paper.
In medical journals, though, the problem is worse because it is more likely to kill someone.
A lot of the medicine paper fraud is in Psychology papers. They are notorious for badly setup studies with intentionally leading questionaires for data. No amount of peer review seems to improve them and a journal somewhere will always publish it.
The volume is quite staggering as well, ~40 Psychologists are responsible for an insane amount of papers all with the same methodology problems showing it can cure everything. Apparently every disease has Psychology problems. In practice no one is getting better from their proposed and trailed treatments and patients repeatedly complain about it, but they are everywhere in Europe and the vast majority of doctors believe in these specialist even though the papers are universally review as very low quality even when they aren't faking data or manipulating the stats to produce an outcome.
There is a reproducibility (replication) crisis in psychology. Much of what psychologists accepted as settled for years has turned out to be bunk. While there are some researchers doing excellent work, much of the field remains no more scientific than phrenology.
Im advocating for "You keep what you kill" rules in science.
If you disprove a paper or proof a study can not be replicated, you get the funds of the scientist, subtracted from his/her current funding. Make bad science fund the good science and make de-replication a for profit endeavor. There can be all funding in the world for quack science, but if it can be debunked and is debunked, it will finance real science.
As a thought experiment I think that’s brilliant. As an actual plan though I think it would be terrible. There are so many ways to fail to reproduce an experiment. Incentivising the failure to reproduce is just a very out there idea.
Since we're on the topic of bad science, the first figure in that article (Pants on Fire) is a pretty bad one. The units on the Y-axis are not specified (strike one), but it looks to be the _unnormalized_ number of papers retracted each year.
The yearly retraction _rate_ (i.e. the number of retracted paper per number of papers published) is what is relevant here. The number of journals have and papers have exploded since '96.
Richard Smith was the editor of The BMJ until 2004.
Competing interest: RS was a cofounder of the Committee on Medical Ethics (COPE), for many years the chair of the Cochrane Library Oversight Committee, and a member of the board of the UK Research Integrity Office.
If you look at why papers are found out as fraudulent and retracted it's usually very dumb mistakes, such as the examples from the article, copying data and/or text from elsewhere, making up numbers that are obvously implausible, and cloning/photoshopping figures or parts thereof, even within the paper. Given how easy it is to avoid these beginner mistakes the percentage of fake data must be so much higher than the actual retractions. Especially from the paper mills where faking data is done by professional fraudsters.
This looks like a good opportunity to ask a question I've had for a long time. In what field(s) of research is it respectable to ask, "Has this paper been peer reviewed?"
I've spent nearly four decades working mostly in academia with scientists, engineers, social scientists of various degrees of rigor, and even occasional humanities types. I don't think I have ever been in a group where someone did or would raise the issue of peer review when discussing the quality of a report.
In all groups I've associated with, you are expected to know enough about your field to be able to assess for yourself the quality of a paper as you read it. Relying on some anonymous reviewer's judgement to justify your acceptance of a report would cast serious doubt on your own judgement.
I've been trying to note just who raises this issue and they appear to be most often in life sciences/medicine. I've never worked in that general area, but my impression is that knowledge of statistics and methodology is rather weaker than in other scientific fields. My (perhaps uncharitable) theory is that peer review has become a gateway for automatic acceptance because the average technical expertise of readers is so low that they cannot evaluate for themselves.
I used to follow a blog (possibly retraction watch? I can't remember now) where they would go through basic biology papers and find photoshopped/edited images and ask the author about them. Sometimes this would lead to retractions etc.
They did mention other people had tried and failed to build tools for this, but the current state of the art was drinking-a-coffee-and-looking-at-it-real-hard. This was before the current explosion of AI, so maybe it's different now?
I've talked about an experience someone in my family had before. Straight up fabrication in a lab, and it wouldn't be really detectable until you attempt to replicate and then check the pictures after.
For the reasons you say, I regard papers that report p-values without effect sizes to be at most interesting, but probably irrelevant for making actual real-life decisions.
I think the one piece people miss is that scientists dont assume the literature is reliable.
When i worked in the lab, it was well know that a good part of the literature can not be replicated.
On top of that, specific journals and labs have reputations for less-than-quality research. If a paper comes out, its not assumed to be true until proven otherwise (usually expanded upon by others). The only time that's different is if it comes from certain labs that are known to have a good track record.
So its not like shoddy research is causing other scientists to waste time except maybe the time spent reading the paper.
I wonder if journals, especially medical ones, could be sued if they don't react and redact studies in time once notified of problems, and patients get ineffective or even detrimental treatment based on those papers.
96% vaccine effectiveness! Into absolute uselessness when deployed to the masses.
As someone who believed them and took the vaccine, that was the last time I'm trusting medical research. I won't be fooled again, and from now on research won't be enough to make any personal decision, I'm waiting until independent evaluation in the real world happens and there are enough anecdotes to see that it really works as advertised.
The entire coronavirus debacle has corroded the trust in medical establishment to zero. I think it's justified and we've just been ignorant to the fraud, it took a lie so big and obvious to really wake people up.
[+] [-] dadrian|3 years ago|reply
[+] [-] aggie|3 years ago|reply
Here is an argument that peer review is basically a failed experiment: https://experimentalhistory.substack.com/p/the-rise-and-fall...
[+] [-] jltsiren|3 years ago|reply
A research paper is supposed to be a honest report of best efforts to study a topic. If it's not, that's a problem that can't be solved by having a few people spending a few hours with the report. The paper is not the final word on anything anyway. If you read a paper expecting to learn something about the world, you are doing something wrong.
As a reviewer, you determine whether the paper is interesting and relevant to the venue. You report any issues you spot that should be corrected and any things you believe that could be improved. And if you get any ideas you feel like sharing, you may share them as well. And that's it.
[+] [-] lr1970|3 years ago|reply
I beg to differ. Peer review is an adversarial process. The author(s) of the paper are making statements and report findings proving them by logic or/and data and analysis. The priors while reviewing a science paper is "wrong" until proven "correct". This also covers accuracy and veracity of the data and analysis. The only thing a peer review is not is assigning intent or blame. It is not the job of a reviewer to look for fraud when simple incompetence could explain it. But after reading this article I will add fraud dimension to a list of fallacies I am looking for while peer-reviewing a manuscript.
[+] [-] NtochkaNzvanova|3 years ago|reply
1. Validity of the experiment
2. Interestingness / novelty of the work
3. Appropriate choice of methods and correctness / believability of the results
4. Signs of outright fraud (suspicious figures, etc.)
Absolutely if I notice some weird Photoshop artifacts in a figure, or some other obvious sign of fake data, I'm going to call that out. I'm probably an outlier on this next one, but I've even been tempted to reject articles just for having egregiously, unreadably bad writing. I know this will be regarded as bias against people whose first language is not English, but if the writing is so bad that it stands in the way of making sense of the article, and the authors can't be bothered to get a decent editor, it's not a worthy contribution to the (English language) scientific literature.
[+] [-] apienx|3 years ago|reply
I'd say these are underestimates. Let me add that 80%+ of papers are useless. The only "value" they provide is to the person getting academically promoted and/or building their publishing portfolio/cred.
[+] [-] ska|3 years ago|reply
As far as I can see this is mostly an incentives problem. Bureaucratic control of academic hiring has ended up emphasizing the short term measurable (e.g. #of pages published, so-called impact factors, etc.) over the long term, with pretty predictable results.
Medical research in particular is fraught with another set of problems; the default clinical pathway gives a weak at best grounding in science, and even the MD/PhD programs have been gamed to some degree. There are definite counterexamples (lots!) but there are also a lot of clinicians with incentive to produce research but little skill in it and even less time available...
[+] [-] nradov|3 years ago|reply
[+] [-] throwawaysleep|3 years ago|reply
[+] [-] parton|3 years ago|reply
People assume that peer review means a paper is good, which couldn't be farther from the truth. Science journalists aren't any better, they care more about hype than consensus. Honestly, it's dangerous to give a random peer reviewed article to someone who doesn't have broad knowledge of the field.
Maybe we need middle-ground journals that publish review articles at the level of a Scientific American reader?
[+] [-] pella|3 years ago|reply
"Conclusions: Parachute use did not reduce death or major traumatic injury when jumping from aircraft in the first randomized evaluation of this intervention."
https://www.bmj.com/content/363/bmj.k5094
see the "Peer review" https://www.bmj.com/content/363/bmj.k5094/peer-review
[+] [-] ramraj07|3 years ago|reply
And did I mention my advisors released a drug into the market just recently? Lol.
[+] [-] brnaftr361|3 years ago|reply
[+] [-] epgui|3 years ago|reply
[+] [-] AlotOfReading|3 years ago|reply
1. Sometimes the latest article on a subject was written 2/5/10 years ago. Depending on quickly the field is moving, that review could be perfectly acceptable, completely obsolete, or anywhere in between.
2. Sometimes the author themselves is just out of date with the field. It's difficult to identify this without deep prior familiarity.
3. Paywalls.
I don't know what the solutions are beyond "making experts accessible for questions" and other public outreach things.
[+] [-] Gatsky|3 years ago|reply
[+] [-] i-use-nixos-btw|3 years ago|reply
In medical journals, though, the problem is worse because it is more likely to kill someone.
[+] [-] PaulKeeble|3 years ago|reply
The volume is quite staggering as well, ~40 Psychologists are responsible for an insane amount of papers all with the same methodology problems showing it can cure everything. Apparently every disease has Psychology problems. In practice no one is getting better from their proposed and trailed treatments and patients repeatedly complain about it, but they are everywhere in Europe and the vast majority of doctors believe in these specialist even though the papers are universally review as very low quality even when they aren't faking data or manipulating the stats to produce an outcome.
[+] [-] nradov|3 years ago|reply
https://www.theatlantic.com/science/archive/2018/11/psycholo...
[+] [-] hn_version_0023|3 years ago|reply
Right, with psychology, you're more likely to kill yourself
[+] [-] PicassoCTs|3 years ago|reply
If you disprove a paper or proof a study can not be replicated, you get the funds of the scientist, subtracted from his/her current funding. Make bad science fund the good science and make de-replication a for profit endeavor. There can be all funding in the world for quack science, but if it can be debunked and is debunked, it will finance real science.
[+] [-] LeonB|3 years ago|reply
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] bobmaxup|3 years ago|reply
[+] [-] avs733|3 years ago|reply
[+] [-] pella|3 years ago|reply
[+] [-] ubj|3 years ago|reply
[1]: https://retractionwatch.com/
[+] [-] Yorch|3 years ago|reply
[+] [-] jszymborski|3 years ago|reply
The yearly retraction _rate_ (i.e. the number of retracted paper per number of papers published) is what is relevant here. The number of journals have and papers have exploded since '96.
[+] [-] tchvil|3 years ago|reply
Richard Smith was the editor of The BMJ until 2004.
Competing interest: RS was a cofounder of the Committee on Medical Ethics (COPE), for many years the chair of the Cochrane Library Oversight Committee, and a member of the board of the UK Research Integrity Office.
[+] [-] breck|3 years ago|reply
But have hope, we're gonna fix it: https://cancerdb.com/
[+] [-] shellfishgene|3 years ago|reply
[+] [-] wrp|3 years ago|reply
I've spent nearly four decades working mostly in academia with scientists, engineers, social scientists of various degrees of rigor, and even occasional humanities types. I don't think I have ever been in a group where someone did or would raise the issue of peer review when discussing the quality of a report.
In all groups I've associated with, you are expected to know enough about your field to be able to assess for yourself the quality of a paper as you read it. Relying on some anonymous reviewer's judgement to justify your acceptance of a report would cast serious doubt on your own judgement.
I've been trying to note just who raises this issue and they appear to be most often in life sciences/medicine. I've never worked in that general area, but my impression is that knowledge of statistics and methodology is rather weaker than in other scientific fields. My (perhaps uncharitable) theory is that peer review has become a gateway for automatic acceptance because the average technical expertise of readers is so low that they cannot evaluate for themselves.
[+] [-] pedalpete|3 years ago|reply
[+] [-] anitil|3 years ago|reply
They did mention other people had tried and failed to build tools for this, but the current state of the art was drinking-a-coffee-and-looking-at-it-real-hard. This was before the current explosion of AI, so maybe it's different now?
Anyone else able to chime in?
Edit: Found it! https://scienceintegritydigest.com
[+] [-] eep_social|3 years ago|reply
[+] [-] renewiltord|3 years ago|reply
https://news.ycombinator.com/item?id=25926188
[+] [-] corbulo|3 years ago|reply
Its difficult to trust really almost any study even if you find parts of it to be reliable.
Take one or a few stats courses to find out how easy it is to smudge data with no one being the wiser. Its a real problem.
[+] [-] krona|3 years ago|reply
[+] [-] refurb|3 years ago|reply
When i worked in the lab, it was well know that a good part of the literature can not be replicated.
On top of that, specific journals and labs have reputations for less-than-quality research. If a paper comes out, its not assumed to be true until proven otherwise (usually expanded upon by others). The only time that's different is if it comes from certain labs that are known to have a good track record.
So its not like shoddy research is causing other scientists to waste time except maybe the time spent reading the paper.
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] shellfishgene|3 years ago|reply
[+] [-] machina_ex_deus|3 years ago|reply
As someone who believed them and took the vaccine, that was the last time I'm trusting medical research. I won't be fooled again, and from now on research won't be enough to make any personal decision, I'm waiting until independent evaluation in the real world happens and there are enough anecdotes to see that it really works as advertised.
The entire coronavirus debacle has corroded the trust in medical establishment to zero. I think it's justified and we've just been ignorant to the fraud, it took a lie so big and obvious to really wake people up.