[dabreegster]

In my experience, the published paper is super vague on the approach, and implementing it without further references is really hard. I'm not necessarily arguing that papers should get longer and more detailed to counter this; expressing the details that matter in code seems like a more natural way to communicate anyway.

Why trust results if you can't see the methodology in detail and apply the approach to your own data? I once knew somebody who built a fuzz tester for a compilers project, got ahold of a previous project's compiler code that won best paper awards, and discovered a bunch of implementation bugs that completely invalidated the results.

Why is the peer review process limited to a handful of people who probably don't have access to the code and data? If your work is on Github, anybody can come along and peer review it, probably in much more detail. And as a researcher, you don't get just one chance to respond to their feedback -- you can actually start a dialogue, which other people are free to join in.

As long as a project's README makes any sort of quality / maintenance expectations clear upfront, why not publish your code?

[anon_123g987]

> In my experience, the published paper is super vague on the approach, and implementing it without further references is really hard.

This is my experience, too, and in my opinion this is exactly what has to change for really reproducible research, not ready to run software supplied by the author.

There are many good arguments in support of publishing code, but reproducibility is not one of them, that's all I'm saying.

[dr_zoidberg]

And just like OP said, it generally takes a couple of months to go from paper to working code. I've implemented a few papers as code as a side-gig for a while, and I wouldn't mind having a baseline from the authors to check and see if I'm following the paper correctly!