Can you provide an example of where this has been successful?
I've spoken with many researchers and grad students only to find that there was a critical typo in the algorithm or undescribed setting (e.g., only converges when a learning rate scheduler is applied) in the code. I'll see the same algorithm implemented differently in different repositories. This is even the case for papers I've found with thousands of citations. It can be tremendously difficult to reproduce the results of papers, especially when they may require large amounts of compute that small researchers don't have. And I don't know whether it's a bug in my code or the paper or the algorithm.
bravura|2 years ago
techwizrd|2 years ago
I've spoken with many researchers and grad students only to find that there was a critical typo in the algorithm or undescribed setting (e.g., only converges when a learning rate scheduler is applied) in the code. I'll see the same algorithm implemented differently in different repositories. This is even the case for papers I've found with thousands of citations. It can be tremendously difficult to reproduce the results of papers, especially when they may require large amounts of compute that small researchers don't have. And I don't know whether it's a bug in my code or the paper or the algorithm.