[shadowsun7]

I should note that this essay kicks off an entire series that eventually culminates in a detailed examination of the Amazon Weekly Business Review (which takes some time to get to because of a) an NDA, and b) it took some time to test it in practice). The Goodhart’s Law essay uses publicly available information about the WBR to explain how to defeat Goodhart’s Law (since the ideas it draws from are five decades old); the WBR itself is a two decades-old mechanism on how to actually accomplish these high-falutin’ goals.

https://commoncog.com/the-amazon-weekly-business-review/

Over the past year, Roger and I have been talking about the difficulty of spreading these ideas. The WBR works, but as the essay shows, it is an interlocking set of processes that solves for a bunch of socio-technical problems. It is not easy to get companies to adopt such large changes.

As a companion to the essay, here is a sequence of cases about companies putting these ideas to practice:

https://commoncog.com/c/concepts/data-driven/

The common thing in all these essays is that it doesn’t stop at high-falutin’ (or conceptual) recommendation, but actually dives into real world application and practice. Yes, it’s nice to say “let’s have a re-evaluation date.” But what does it actually look like to get folks to do that at scale?

Well, the WBR is one way that works in practice, at scale, and with some success in multiple companies. And we keep finding nuances in our own practice: https://x.com/ejames_c/status/1849648179337371816

[skmurphy]

It looks like any other decision record where you set a date to evaluate the impact of a policy or course of action and make sure it's working out the way that you had anticipated.

[shadowsun7]

And how are you going to tell that when you have a) variation (that is, every metric wiggles wildly)? And also b) how are you able to tell if it has or hasn’t impacted other parts of your business if you do not have a method for uncovering the causal model of your business (like that aquarium quote you cited earlier?)

Reality has a lot of detail. It’s nice to quote books about goals. It’s a different thing entirely to achieve them in practice with a real business.