[jostmey]

This only addresses a small part of the problem with peer-review. The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers. If the paper doesn’t fit what is expected, it will often be rejected. This can also lead to self-reinforcing views that ignore contrarian data. Also, the data can be made up, and if it makes sense to the reviewers, it is generally not questioned

[kergonath]

> The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers.

This is a misunderstanding of the role of peer review. The point is not to prove that a paper is correct, the point is to ensure a minimum level of quality. You are entirely right that most reviewers cannot hope to reproduce the results presented, and very often for very good reasons. If I write 3 proposals over the course of a year to get some beam line on a neutron source, it is completely unrealistic to expect a referee to have the same level of commitment.

I think this hints at a more profound problem, which is that a lot of studies are not replicated. This is where the robustness of a scientific result comes from: anybody can make the same observation and reach the same conclusions under the same conditions. This is the real test, not whether you convinced 3 or 5 referees.

The real value of an article is not in whether it was peer reviewed (though the absence of peer review is a red flag). Instead, it is in whether different people confirmed its main results over the years that follow publication.

[mike_hearn]

There are no silver bullets. Replication also isn't worth much:

https://blog.plan99.net/replication-studies-cant-fix-science...

Merely replicating a study doesn't tell you anything about whether the claims are correct, although failure to replicate can be a signal that the claims probably aren't correct.

[yummypaint]

Have you personally reviewed a non-zero number of papers? What is this statement based on? For a thread ostensibly about science, the comments are disappointingly lacking in evidence and heavy on vibes.

Maybe people could learn about what peer review is before posting their strong feelings about it? The purpose certainly isn't to replicate people's experiments, that happens after publication and not by referees. One of a reviewer's duties is to look at whether the study could be replicated given the included information. That is a very different thing.

Also, just because something has made it past peer review also doesn't mean it isn't controversial in the field.

[jostmey]

I’ve only peer reviewed a few papers. I’ve submitted my share of papers for review

This is my academic profile https://scholar.google.com/citations?user=bxn78bkAAAAJ&hl=en

[mike_hearn]

He isn't saying the purpose of peer review is to replicate work. And yes I've personally reviewed quite a lot of papers (from outside of academia), also occasionally reviewed peer reviews.

What he is saying is that peer review is treated by academia as a close to gold standard, when it's really more like a bronze standard. It's a bit like finding a software company that exclusively uses volunteer post-commit code review with no unit tests or static typing, and in which the only testing process is to push to production and see if anyone inside the company complains.

It's not useless as a concept, and it's generally better to have a paper that's reviewed than one that isn't unless the field has been captured by ideologues. But there are so many common problems it can't fix, not even in principle.

[aDyslecticCrow]

Bad quality work is a much bigger problem than dishonest work. Systematically well-done research with fake results or data is much rarer than just... lazy bad science.

There is a massive incentive to publish. Inflate the value, inflate the results, and stretch out projects to multiple smaller papers, fake results to make it seem important. This is lazy and fast, and can be caught by a stricter review and scrutiny.

Papers that are properly done all the way through, but with faked data meant to push an agenda, can be disproven by counter research.

[fc417fc802]

> Systematically well-done research with fake results or data is much rarer than just... lazy bad science.

Genuine mistakes, logical errors, and other oversights are even more common than that. For all the issues it has, peer review is quite good at catching the things that it's intended to catch.

[jxjnskkzxxhx]

Reading your comment makes me think that you believe that the point of peer-review is to ensure that a paper is correct, or at least that specific aspects of it are correct. Is that the case? What do you think the point of peer-review is?

[yupitsme123]

I'm not the person you replied to, but I think that in the lay world, people do indeed think that peer review is as you've described. If it's not, then maybe it should be?

Research gets cited constantly in public debates and is used for policy decisions, so the public should be able to quickly separate the good from the bad, the "maybe this is true" from the "this is empirically proven."

The public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus.