[codekilla]

> To expect a national-level organization to essentially take over the duties of peer-review journals is a very big ask

Fair, but what is the alternative that would actually work? What is the budget of all of the journals compared to the NSF+NIH? Is medical research that is true, and certainly actionable worth as much as an F whatever fighter jet? People will have to decide.

[bumby]

The tradeoff is a different arguement and a digression, so while it's likely we agree, I'll side-step it here. I do think it's an uphill battle to pursue a massive reappropriation of funds though.

There are some alternatives that I'm aware of. Here's a few:

1) One is to allow journals to focus on less-than-great results. Right now the focus is on novelty, so there is an incentive to show that your work has some new, great outcome. But there's also value in showing "Hey, we thought this idea had legs but it turns out it didn't." Publishing that work should be part of science but right now its not. (As a side benefit, you could prevent a lot of researchers wasting effort on the same idea simply because they weren't aware that other people already tried, and failed.)

2) Journals can put a premium on sharing your data and code during the review process. Right now, it's often just up to the author and there are lots of veils to hide behind that essentially give the impression of sharing data, but not in a very useful way.

3) Give value to replicating work. Maybe not as much prestige as creating new work, but showing that it can be replicated obviously has value to society as a whole. Most of the time this won't get published, except in the cases where it's sensationalized, like fraud. (This effect is related to #1)

4) Journals can do a better job vetting their reviewers. They struggle to get timely reviews and reach to anyone who accepts the duty. Reviewers may agree to review something they have little background in, and as a result, it's easier to skirt bad articles through the system.

[codekilla]

I don’t disagree with any of these points, it’s just I’ve been involved in open science circles, where these things are always mentioned, and I just don’t see any material progress (maybe I’m not looking that closely though). I think the reason for the lack of progress is mainly funding—so until someone gets serious about funding (billionaires or taxpayers), it just seems like the same merry go around. It’s very expensive to replicate biomedical studies—but it’s the only thing that works. Maybe the tide is turning though and simply incentivizing/protecting grad students to become whistleblowers will do more good, but I fear this case was more the exception than the rule.