[jegp]

Thank you for your seriously thoughtful insights. You're saying we should have reproducibility, communication, and exploration. I fully agree with that, but I would like to push back on your point that novelty is destructive. Can't redundancy be similarly destructive? Given finite resources, shouldn't the authors have some responsibility to check their findings against previous insights? Otherwise, in the worst case, Goodhart can forget about his law because we'll never agree on any metrics in the first place.

To clarify, this is orthogonal to the point about the paper. I'm still not convinced it's entirely different, but I agree that the value of the paper doesn't purely depend on that. It's a nice exposition irregardless.

[godelski]

I'm not sure if you'll be surprised to find out that I agree with you.

Thing is, we're naturally inclined to pursue new things. There's already incredible amounts of natural pressure to pursue progress and innovate. It is much more lucrative regardless of the criteria for publication. Even the most modest of researchers are overjoyed when they find a clever solution or stumble on something no one else has before. We're naturally driven in this direction.

That said, I'm not going to claim this too isn't hackable. All metrics are hackable. My belief is just that metrics are guides, not answers. I believe Goodhart would agree, and that's his point. His point is yours, that no one will agree on metrics in the first place. Because the truth is, that no metric is perfectly aligned with a goal. His law isn't saying how to generate a good metric, his law is a warning about over reliance on metrics. To use them mindlessly.

The thing is you have to embrace the chaos. Maybe consider Pournelle's Iron Law of Bureaucracy as well. Bureaucrats can't create nuanced metrics, they must be mindless or else they would need an expert to interpret them in the first place.