I'll barge in and say that there's this meme in business schools that: "what gets measured, gets done."
They're mostly there to serve as the source of truth, whether or not a specific manager is getting a bonus (and by how much).
Generally, the best metrics are "based in reality" (and usually revolve around inflow and outflow of money) -- but those are hard to fudge, without playing accounting games, and even harder to meet (it takes actual skill, rather than being able to pass the buck endlessly for a couple of years, before moving onto another org); so you may find it common for the managerial horde to pick "bullshit metrics" to cover their own asses (doubly so, if the metrics are decided by committee).
There's no reason for practical metrics to exist in most large corporations -- the incentives just aren't aligned. Bonuses are decided by metrics (and anyone who has a modicum of intuition, will always pick the metric that can be most gamed for himself, and his underlings). Also, if we're being real with ourselves, there's very little that managers "control," so picking a "real" metric would just breed an environment that selects for luck (or lies).
In my opinion, the only useful metric is profit generated. Everything else is just an emotional safety blanket for uncertainty.
> In my opinion, the only useful metric is profit generated. Everything else is just an emotional safety blanket for uncertainty.
This too can be a toxic metric. Only focusing on what looks good in the quarterly report encourages a type of short-sightedness that has been the undoing of many businesses.
The project I've been on for the last couple of years has had dashboards and monitors and we as engineers have been striving to keep the system humming and if any issues trigger alarms, we jump on them.
More recently there's been a push to handle alarms more quickly (get those response times down!). I'm not sure why since I don't know if we've really had user complaints about any of this (not that our systems are perfect, but the user-facing issues aren't due to us not responding quickly enough to alarms). It never made sense to me that these numbers were of utmost importance, and important enough to visit in biweekly status meetings.
Your post now makes me see more clearly what I had a hunch: somebody decided to take on this mission of getting response times down as an action item and by god everyone in their org is going to meet that mission.
Anyway, the product we were working on got cancelled recently, which funnily enough has nothing to do with alarm response time or any of our engineering dashboards and more to do with the "money in/money out" metric. Fancy that.
The only thing that matters is revenue, and if it can't be easily measured against revenue it doesn't exist. I am so tired of this line of reasoning. It's poisoning the industry, ruining products, and sucking the joy out of work.
What about making a quality product because you have a passion for it? Nope, make a broken MVP and move on to the next customer.
If I can't maintain a metric until I can tie it to revenue, then I guess I don't get cpu usage on our database. Or disk usage. Or response time for anything that only employees touch. Bad data and errors are fine as long as no one complains. Actually, if we produce a shitty product long enough everyone will get used to it and stop complaining. If we stop measuring error rates we can stop wasting time on fixing things right? Tests and code refactors just get in the way of delivering that MVP. Rewards are doled out for fixing an outage, and preventing them is a cost-center to be eliminated.
On the flip side, now we've got KPIs for our ability to push things through the corporate bureaucracy. I didn't become a programmer so I can waste time estimating how long it'll take to produce a design doc that'll be useless by the time I get to implementation, just so a bureaucrat can decide if some deeply technical problem they don't understand should be fixed.
> So let's circle back to metrics and ways to ensure we use them for guidance rather than obey them reflexively.
I'm not sure this is the right framing of the problem. "Guidance" is too nebulous and leaves us with the same problem of interpretation.
Advice from an old tracker: You want to find someone, use your eyes.
When I have someone who is getting themselves/us in trouble by looking at charts, my remedy always includes the notion that charts are for asking questions, not answering them. You might have a chart that says user response time has improved, but have you tried using the service from your phone (with wifi disabled)? If the chart says the times are shifting, we need to know why they are shifting, because that's information. "Looks like response time went down," is only useful as the introduction to a question, like, "did we change something?" or "did we have a routing issue?", or "can we do the same thing on this other service that is drawing complaints?"
Getting good questions also often involves cleaning up sources of noise in the charts. It's too easy for someone with an axe to grind to jump to conclusions that this is a repeat of a problem we had before (ie, this is Team X's fault... again) or that lack of improvement means failure to act. It is not impossible that I fix one regression at the same time someone else causes another, or changes the denominator by the same magnitude as the fix.
I don't have much to add other than this reminds me of the obsession around "satisfaction" for product metrics and why that seems to always boil down to slapping NPS on it because it's a number and execs like numbers.
Goodhart's law is often confused with Campbell's law.
The two are indeed very similar, and often used interchangeably, but there's an important difference in their phrasing.
Goodhart's law states that: "When a measure becomes a target, it ceases to be a good measure."
Campbell's law states that: "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."
In other words, the former focuses on the fact that, while metrics sound good in theory, in practice they cannot be relied on; whereas the latter focuses on the fact that enforcing metrics in an attempt to control a social process tends to backfire.
“I'll barge in and say that there's this meme in business schools that: "what gets measured, gets done."” The key is those that sone might not be most useful.
throwaway2331|4 years ago
They're mostly there to serve as the source of truth, whether or not a specific manager is getting a bonus (and by how much).
Generally, the best metrics are "based in reality" (and usually revolve around inflow and outflow of money) -- but those are hard to fudge, without playing accounting games, and even harder to meet (it takes actual skill, rather than being able to pass the buck endlessly for a couple of years, before moving onto another org); so you may find it common for the managerial horde to pick "bullshit metrics" to cover their own asses (doubly so, if the metrics are decided by committee).
There's no reason for practical metrics to exist in most large corporations -- the incentives just aren't aligned. Bonuses are decided by metrics (and anyone who has a modicum of intuition, will always pick the metric that can be most gamed for himself, and his underlings). Also, if we're being real with ourselves, there's very little that managers "control," so picking a "real" metric would just breed an environment that selects for luck (or lies).
In my opinion, the only useful metric is profit generated. Everything else is just an emotional safety blanket for uncertainty.
marginalia_nu|4 years ago
This too can be a toxic metric. Only focusing on what looks good in the quarterly report encourages a type of short-sightedness that has been the undoing of many businesses.
allenu|4 years ago
The project I've been on for the last couple of years has had dashboards and monitors and we as engineers have been striving to keep the system humming and if any issues trigger alarms, we jump on them.
More recently there's been a push to handle alarms more quickly (get those response times down!). I'm not sure why since I don't know if we've really had user complaints about any of this (not that our systems are perfect, but the user-facing issues aren't due to us not responding quickly enough to alarms). It never made sense to me that these numbers were of utmost importance, and important enough to visit in biweekly status meetings.
Your post now makes me see more clearly what I had a hunch: somebody decided to take on this mission of getting response times down as an action item and by god everyone in their org is going to meet that mission.
Anyway, the product we were working on got cancelled recently, which funnily enough has nothing to do with alarm response time or any of our engineering dashboards and more to do with the "money in/money out" metric. Fancy that.
cconstantine|4 years ago
What about making a quality product because you have a passion for it? Nope, make a broken MVP and move on to the next customer.
If I can't maintain a metric until I can tie it to revenue, then I guess I don't get cpu usage on our database. Or disk usage. Or response time for anything that only employees touch. Bad data and errors are fine as long as no one complains. Actually, if we produce a shitty product long enough everyone will get used to it and stop complaining. If we stop measuring error rates we can stop wasting time on fixing things right? Tests and code refactors just get in the way of delivering that MVP. Rewards are doled out for fixing an outage, and preventing them is a cost-center to be eliminated.
On the flip side, now we've got KPIs for our ability to push things through the corporate bureaucracy. I didn't become a programmer so I can waste time estimating how long it'll take to produce a design doc that'll be useless by the time I get to implementation, just so a bureaucrat can decide if some deeply technical problem they don't understand should be fixed.
There has to be a better way.
hinkley|4 years ago
I'm not sure this is the right framing of the problem. "Guidance" is too nebulous and leaves us with the same problem of interpretation.
When I have someone who is getting themselves/us in trouble by looking at charts, my remedy always includes the notion that charts are for asking questions, not answering them. You might have a chart that says user response time has improved, but have you tried using the service from your phone (with wifi disabled)? If the chart says the times are shifting, we need to know why they are shifting, because that's information. "Looks like response time went down," is only useful as the introduction to a question, like, "did we change something?" or "did we have a routing issue?", or "can we do the same thing on this other service that is drawing complaints?"Getting good questions also often involves cleaning up sources of noise in the charts. It's too easy for someone with an axe to grind to jump to conclusions that this is a repeat of a problem we had before (ie, this is Team X's fault... again) or that lack of improvement means failure to act. It is not impossible that I fix one regression at the same time someone else causes another, or changes the denominator by the same magnitude as the fix.
phillipcarter|4 years ago
I think this is essential reading if anyone's interested on a product-focused tangent for the topic: https://articles.uie.com/net-promoter-score-considered-harmf...
malshe|4 years ago
splittingTimes|4 years ago
tpoacher|4 years ago
The two are indeed very similar, and often used interchangeably, but there's an important difference in their phrasing.
Goodhart's law states that: "When a measure becomes a target, it ceases to be a good measure."
Campbell's law states that: "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."
In other words, the former focuses on the fact that, while metrics sound good in theory, in practice they cannot be relied on; whereas the latter focuses on the fact that enforcing metrics in an attempt to control a social process tends to backfire.
Or, in Trump language:
Goodhart = "Sounds good, doesn't work"
Campbell = "Worst trade deal ever"
unknown|4 years ago
[deleted]
a9h74j|4 years ago
Thus, the original article title might be a bit misleading. It is not necessarily about businesses running based upon illusions.
ngcc_hk|4 years ago