Looks like he wasn't kidding, at least for Java shops with untuned JVMs. This was linked from the article and from Dan Luu's work at Twitter. "We spent one day1 building a system that immediately found a mid 7 figure optimization (which ended up shipping). In the first year, we shipped mid 8 figures per year worth of cost savings as a result. The key feature this system introduces is the ability to query metrics data across all hosts and all services and over any period of time (since inception), so we've called it LongTermMetrics (LTM) internally since I like boring, descriptive, names.
I'll add that I worked in systems at a large php shop which didn't have good metrics for utilization. After the adoption of HHVM their utilization dropped enough that they ended up buying extra hardware in the 6 figures. They knew that performance was a lot better but they didn't quantify it. So surprise! All it took to figure that out and start making better decisions was to collect SAR data and dump it to a central log. So, yeah. Boring is good or at least a good place to start.
“When a measure becomes a target, it ceases to be a good measure.”
The problem is that a competitive measure tends to overshadow the utility of the measure (look at anything that starts with a TPC- or the first rotation of a Tesla roadster tire).
So the biggest impact of a measure is immediately after it is introduced, because the market hasn't engineering itself to overfit to that measure.
And that is what Dan's post clearly says, that immediately after he brings up something, there is an actual change in the industry practice that happens.
> it doesn't show off creativity.
But something like Jepsen is amazingly useful because it is a creative nothing-up-my-sleeves end-run around first-party testing - actual third party testing for the product is the red-team to a QE team doing the blue-team work. The problem is that a lot of people leave that to the same vendor who is selling the product with its promises from their sales team - it is hardly an open book into the bugs open list (I work in open-source, so I have to wash my dirty laundry in public).
As a side not to this, I had a meeting in the last 2 days where someone told me that the "QE team is doing a good job, because the test have been consistently green as the release approaches".
Most folks don't naturally like to measure things. Most people view the world through their own personal lens and they categorize phenomena through their experiences. It makes it hard to "sell" the idea of measurement, since most people only see the things they're predisposed to see.
Additionally it's hard work. Creating the right tests, getting the data into the format you need to do analysis, collating it and finding the important things, explaining it.
"When the reporter doesn't have a repro for the bug, which is quite common when it comes to distributed systems, the bug will be written off as non-existent"
This.
If this is your modus operandi, then please think twice.
For me, as the person with the problem, this just sounds like "I don't want to be bothered with this right now".
Most maintainers don't see this as a possibility to improve their project, but as trouble for their brilliant idea. Grow up!
I think you have to swim upstream in this, which is to say resist the urge to write systems with complex emergent behavior in the first place. They’re thrilling to write, but even you will get tired of your bullshit at some point and lose interest (often before you figure out why you don’t like working on the code).
I am watching a coworker begin to discover this right now. He’s one of the last standing of a cohort of overconfident loudmouths. It’s been... interesting. No self-awareness so far yet though.
vajrabum|4 years ago
https://danluu.com/metrics-analytics/
I'll add that I worked in systems at a large php shop which didn't have good metrics for utilization. After the adoption of HHVM their utilization dropped enough that they ended up buying extra hardware in the 6 figures. They knew that performance was a lot better but they didn't quantify it. So surprise! All it took to figure that out and start making better decisions was to collect SAR data and dump it to a central log. So, yeah. Boring is good or at least a good place to start.
amadeuspagel|4 years ago
gopalv|4 years ago
Goodhart's law has been a problem with measures.
“When a measure becomes a target, it ceases to be a good measure.”
The problem is that a competitive measure tends to overshadow the utility of the measure (look at anything that starts with a TPC- or the first rotation of a Tesla roadster tire).
So the biggest impact of a measure is immediately after it is introduced, because the market hasn't engineering itself to overfit to that measure.
And that is what Dan's post clearly says, that immediately after he brings up something, there is an actual change in the industry practice that happens.
> it doesn't show off creativity.
But something like Jepsen is amazingly useful because it is a creative nothing-up-my-sleeves end-run around first-party testing - actual third party testing for the product is the red-team to a QE team doing the blue-team work. The problem is that a lot of people leave that to the same vendor who is selling the product with its promises from their sales team - it is hardly an open book into the bugs open list (I work in open-source, so I have to wash my dirty laundry in public).
As a side not to this, I had a meeting in the last 2 days where someone told me that the "QE team is doing a good job, because the test have been consistently green as the release approaches".
And I didn't know exactly how to respond to that.
Karrot_Kream|4 years ago
danielmarkbruce|4 years ago
_tom_|4 years ago
shele|4 years ago
dang|4 years ago
vincnetas|4 years ago
https://danluu.com/
k__|4 years ago
This.
If this is your modus operandi, then please think twice.
For me, as the person with the problem, this just sounds like "I don't want to be bothered with this right now".
Most maintainers don't see this as a possibility to improve their project, but as trouble for their brilliant idea. Grow up!
hinkley|4 years ago
I am watching a coworker begin to discover this right now. He’s one of the last standing of a cohort of overconfident loudmouths. It’s been... interesting. No self-awareness so far yet though.