[jpfr]

I happen to be an (associate) editor of an academic journal.

What the people who critique the publication process are missing: 90% of submissions are crap - unfit for publication.

We need some process to gate-keep.

a) The venue of publication is a good signal, whether the time to read a paper is well-spent. b) The PhD students learning the craft need objective feedback. The supervising professior/university often has the incentive to "just submit" -- even if they know that a publication does not meet the quality standard.

Before peer-review, somebody also needed to make a decision on what to publish. This typically fell to a single individual. The editor or some well-known member of the community who could recommend a paper for publication. On old journal issues they even mention the "recommender".

So the question is not whether peer-review is bad, the question is which alternative gate-keeping process would be better. Otherwise we will drown in crap publications (even more) and the PhD students don't get a honest feedback signal upon which they can improve their craft.

[cherryteastain]

As a (former) reviewer at 5 journals, I disagree first and foremost with the notion that

> We need some process to gate-keep.

Journals, when print was the medium through which academic research was disseminated, had to gatekeep because there were practical considerations regarding how many articles they could put in each issue. With online repositories like arxiv, this is hardly a concern anymore.

Someone putting a crap article on arxiv does not hurt anyone else, and I'm saying this as a person who recommended tons of articles to be rejected because they had atrocious grammar/spelling issues. Worst case, it gets 0 attention and is ignored by the research community.

Something not being published in a journal/conference proceedings clearly does not prevent it from drawing tons of research attention, as we saw in numerous cases like the Adam optimizer [1].

Which brings us to the second point: what even is the purpose of a journal now? The answer is that the sole function of a journal now is gatekeeping, with the presupposition that, as you observed

> The venue of publication is a good signal, whether the time to read a paper is well-spent

Except, well, top journals have tons of articles that get 0 citations too. Clearly the filter fails at this purpose as well. So, why gatekeep at all then? Because if we did not have some exclusive prestigious journal, the plebs would not be separated from the esteemed titans of academia with the biggest grants, most prestigious scholarships and diplomas from the most famous universities.

The only reason we need to gatekeep today is to feed the academic prestige and politics machine. If you care about the science, upload the goddamn PDF to arxiv , tell your colleagues about your research at a conference and let the scientific community decide on whether your idea is interesting.

[1] https://arxiv.org/abs/1412.6980

[Al-Khwarizmi]

The Adam optimizer was published at ICLR, a top conference in machine learning. So, basically, analogous to a peer-reviewed journal for the purposes of this discussion. ML (and some other subfields of computer science) have the particularity that the really comptetitive gatekeeping happens even more in conferences than in journals.

That said, there definitely are very relevant papers that are not published in any peer-reviewed venue. A good example is "Language Models are Unsupervised Multitask Learners" (the GPT-2 paper, which I would argue started the whole generative LLM revolution). But I think if you look for this kind of papers, you will find something in common to all of them: they are by very well known researchers, elite institutions or influential companies. That's why people went out of their way to read them even if they were posted somewhere without peer review.

If you removed peer review and just relied on posting to arXiv or similar, new researchers, or researchers from less known institutions, would have no chance at all to make an impact. It's peer review that allows them to be able to submit to a top journal, where the editor and reviewers will read their paper, and they can get a somewhat fair chance.

PS: I don't really like the peer review system that much either. It's just that every alternative that I have seen proposed so far is worse.

[dopu]

It seems to me that something like eLife's model is the best solution to this [0]: you still have a minimal amount of curation, but generally if a paper is written well enough and within the field it won't be desk rejected. Then, it gets published on the site and sent off for peer review. Peer reviewers assess how sound the paper is and pass a judgement which readers can view, as well as provide some recommendations to the authors make it stronger. The authors can then either revise the paper, or do nothing at all. In either case, papers don't languish in reviewer hell and the larger scientific community gets to see it.

[0] https://elifesciences.org/inside-elife/66d43597/elife-s-new-...

[jpfr]

Exactly, everybody can publish on arXiv. And there are enough semi-predatory journals/conferences which basically accept everybody. Especially since the LLM can rewrite any paragraph nicely.

So the role of journals and conferences is not to prevent "the word getting out". It is to provide value by a curated list of on-topic and high-quality publications.

So no need to wade through tons of crap. Especially for PhD students who might take more time to detect crap as such.

In my experience, the publications at the good venues get a lot more eye-balls and by consequence citations. So there seem to be a lot of people who like this role as a "filter" for what to focus on.

[xipho]

> Someone putting a crap article on arxiv does not hurt anyone else

> ... only reason we need to gatekeep today is to feed the academic prestige and politics machine

This to me says you have may not have experienced some parts of the (long-term) research process. It suggests that you have infinite resources to filter out noise, which is probably not the case. It suggests you're willing to spend a lot more time figuring out why something doesn't quite make sense, rather than get to the heart of the problem, while this is fine in many cases it sucks when you're hot on the trail of something interesting, and you're slammed by a million twisty paths full half-baked hot-takes.

We need to filter ("gate-keep" is pretty inflammatory term) information and processes so that we don't have 12 different screw types with 12 different electric screwdrivers, instead of "just" 6 (sigh). We need to come to consensus and that means some things go in and something are left out. We need many mechanisms to filter.

> tell your colleagues about your research at a conference and let the scientific community decide on whether your idea is interesting.

All of these things feel like filters, when does a filter become a gate: colleagues - i.e. not everybody, but some selected few, who and how?; conference - filter (well, gate!); scientific community - != your baker's community; decisions directed by you, not on my own (i.e. a pointer to my paper) - filter.

[Edit for formatting, sort of.]

[thfuran]

>Someone putting a crap article on arxiv does not hurt anyone else,

But having a browsable collection of the verified non-crap articles on any given topic helps most everyone working in that area.

[selimthegrim]

PNAS still mentions the recommender.

[naasking]

> What the people who critique the publication process are missing: 90% of submissions are crap - unfit for publication.

Publication is antiquated. HN doesn't need reviewers to boost the best content or to provide commentary on how to improve a paper or fact check its contents. Join the 21st century.

[Spacecosmonaut]

We are on the cusp of being able to plow through vast quantities of literature and data in an instant using multimodal machine learning models. Journal articles are written for people. The landscape is changing. We are headed toward a future where scientist upload data and thoughts en masse into the cloud to be consumed by interpreter models that in turn feed back into the scientific machine. Data quality and attribution (scientist performance rating) is automatically allocated by models.