[aggie]

In practice, peer review isn't even that. Most referees are not double-checking your statistical analysis. What they focus on is whether the research is interesting and has it appropriately considered relevant literature. Even that is not always done carefully.

Here is an argument that peer review is basically a failed experiment: https://experimentalhistory.substack.com/p/the-rise-and-fall...

[dadrian]

I think we're saying the same thing. You're trusting the statistical analysis done by the author. When I peer review, I check to make sure that the conclusions the paper draws are in line with what it says statistical analysis was, and I check to make sure that the analysis used is the right one, but I don't check their math. And also, I'm not a statistician, so I'm not authoritative on the full space of statistical analysis, anyway.

[nieto915]

who really has all of the hours in a day to do it too. I think it'll all come down to the journal's integrity. academia will have to learn to negotiate the premiums of their library subscriptions for better editing/curation for each journal they subscribe to. its already pay to play, might as well get your money's worth.

[fsckboy]

In practice, peer review isn't even that. As fields of science have grown more specific, qualified reviewers are peers who are your competition, where you are already in their club or somebody who needs to be kept out. You are not going to be helped in any science that threatens their funding. There are a lot of interests vested in every instance of medical research fraud.

[ouid]

Like many human inventions, it started out good and got worse as people got better at exploiting the flaws in the institution. Those exploits eventually get bad enough that they can escalate to creating new flaws to be exploited until the whole thing is captured. Most institutions are somewhere along that path.

My personal philosophy about such things is to think of Hanlon's razor as a boundary condition. The longer an institution has been around, the more likely the incompetence is actually just well disguised malice.

[wolverine876]

It's imperfect, but what better system do we replace it with? It's easy to critique; it's hard to solve problems. Just like it's easy to find problems in any research, but hard to research and publish.

[anonylizard]

The best system we can currently observe, is in ML.

Look at stable diffusion as an example. Incredible papers, such as dreambooth, LORA, controlnet, are:

1. Published on arxiv before peer review

2. Productionised within 2 weeks of paper release (peer review not needed)

3. Community rapidly adapts tool, makes it easier to use.

4. Products built on such papers proliferate extremely rapidly within another few weeks.

In this system, peer reviews are worthless. The github code quickly demonstrates whether a technique is useful or not, and the community adoption rates replace citations as proof of a paper's power.

This is why AI art can progress at such insane rates, weeks from paper release to widespread productionisation.

Obviously, this won't work in most other domains, because there's no equivalent to mass consumer interest, open source communities, and low-cost experiments. But it does represent the ideal of an academic research paradigm.

[jostmey]

The peer-review system is antiquated, developed before the internet and powerpoint presentations. Anything that facilitates interaction between authors and other researchers will be a vast improvement.

Peer-review doesn't catch fraud and is sometimes a political process. I've found the best corrections I've gotten is after posting pre-prints to online forums. I suggest that commenting on pre-prints is better than peer-review pre-publication. I imagine a ranking system could highlight comments from trusted reviewers. Studies that no one wants to review were probably never going to be read anyway, and so there was never any reason to review these studies anyway.

[BurningFrog]

Widespread peer review is only ~50 years old.

Science did just fine before it, as it will after it's phased out.

[naasking]

> It's imperfect, but what better system do we replace it with?

Pre-registered trials and/or arxiv + open science.

[TheSpiceIsLife]

Science theatre