You'd be surprised how many serious tech shops have close to zero performance metrics collected and utilised.
I've done this in fintech a few times already and the best stack that worked from my experience was telegraf + influxdb + grafana.
There are many things you can get wrong with metrics collection (what you collect, how, units, how the data is stored, aggregated and eventually presented) and I learned about most of them the hard way.
However, when done right and covering all layers of your software execution stack this can be a game changer, both in terms of capacity planing/picking low hanging perf fruit and day to day operations.
Highly recommend giving it a try as the tools are free, mature and cover a wide spectrum of platforms and services.
I'd like to followup on this and say InfluxDB is fantastic -- with the caveat that you know exactly what you want, and use it with that in mind. It's great at storing some metrics with high volume, but I've found the OSS version falls over hard in some circumstances (high cardinality in tags absolutely murders performance, making it nearly unusable).
I'm in the middle of doing a test-run of vmagent + victoria-metrics as a mostly-replacement (see: not replacing the cases where it's known exactly what's wanted, and used explicitly for that), and victoria-metrics fully support telegraf pushes which is a big bonus.
Also they're apparently going to be dropping InfluxQL entirely, for Flux, which is weird to me? The docs don't outright state this, but InfluxDB 2.0 has only Flux examples in the docs I can find. It's a better language, however the learning curve is not trivial.
Agreed; same exp (w many years of web perf optimization as ui arch and/or perf strategy consulting gigs. Even after all these years, simply using WebPageTest to capture test runs and share the results is often a transformational experience.
"I don't have anything against hot new technologies, but a lot of useful work comes from plugging boring technologies together and doing the obvious thing."
Starting the article off with "I did this in one day" - complete with a massive footnote disclaiming that it obviously took a lot more than one day - kinda ruined it for me. Why even bother with that totally unnecessary claim?
My read on it is the author is saying that seemingly small changes can have big impacts. I agree it could have been worded better, though I doubt he's trying to promote himself as a genius (as other people are saying) because he clearly highlights the effort his team put into the project in the footnote.
"I did it by myself in one day, well actually it was one week but had I known the stack I would likely have done it in one day. Oh, and by the way, after that week, there was yet another month of work involving at least two other persons from my team and then even more work from other teams. But let's not dwell on boring details".
It's nearly as infuriating as the "Appendix: stuff I screwed up" which doesn't contain actual screw up. It's a shame because the rest of the writing is interesting and doesn't need to be propped up.
It's kind of a personal marketing thing these days to have this maverick/hero aura of genius instead of the "unproductive" but real and hard grinding work to get something done and delivered. It worked for a few so thousands try the same and we are here now, I guess?
That strikes me as a short tolerance for feeling something is ruined. He appropriately highlighted the real time
estimate of the more involved work in a footnote. He didn't literally mean all of the work was one day, he's trying to convey a larger point about outsized engineering returns from comparatively small person-hours of work.
Were you able to move past this to read the rest of the article? Because it's a very good article.
There's a lot of interest in this space with respect to analytics on top of monitoring and observability data.
Anyone interested in this topic might want to check out an issue thread on the Thanos GitHub project. I would love to see M3, Thanos, Cortex and other Prometheus long term storage solutions all be able to benefit from a project in this space that could dynamically pull back data from any form of Prometheus long term storage using the Prometheus Remote Read protocol:
https://github.com/thanos-io/thanos/issues/2682
Spark and Presto both support predicate push down to a data layer, which can be a Prometheus long term metrics store, and are able to perform queries on arbitrary sets of data.
Spark is also super useful for ETLing data into a warehouse (such as HDFS or other backends, i.e. see the BigQuery connector for Spark[1] that could write a query from say a Prometheus long term store metrics and export it into BigQuery for further querying).
I was thinking the answer to 'why are programmers paid more than other white-collar professions that are similarly profitable' is: because programmers control the means of production.
I might be a great telecomm tech, a genius even, but once I'm out of a job, I can't build my own telecomm system - that would cost billions. I have to go back to some other telecomm system to start making money again.
But, at a startup, a kicked-out senior engineer can actually pretty much exactly recreate the company; they can do the equivalent of a laid-off telecomm employee starting a new, almost-as-good (except for branding) telecomm company.
No billions in infrastructure required: within a month or two, the cloned company could be near-indistinguishable from the original.
So companies have to pay employees more like partners, instead of employees, because either they pay them as equals or they'll be forced to compete against them, as equal rivals.
Just so I understand, the simple way the headline talks about was "collect all metrics, but store the anal fraction you care about in an easily accessible place; delete the raw data every week"?
Title didn't live up to article imho. But I get it. Thanks for sharing your methods.
Love the section in this about using "boring technology" - and then writing about how you used it, to help counter the much more common narrative of using something exciting and new.
If we consider Graphite, InfluxDB, and Prometheus, at this point in the monitoring industry's evolution, we have the capability to easily criss-cross metrics generated in the format of one of these systems to store them in one of the other ones.
The missing piece remains to be able to query one system with the query language of the others. For example, query Prometheus using Graphite's query language.
Speaking of high cardinality metrics, what are good options that aren’t as custom as map reduce jobs and a bit more real time?
We killed our influx cluster numerous times with high cardinality metrics. We migrated to Datadog which charges based on cardinality so we actively avoid useful tags that have too much cardinality. I’m investigating Timescale since our data isn’t that big and btrees are unaffected by cardinality.
The boring technology observation (here referring to the challenge of getting publicity for "solving a 'boring' problem") is really true.
It extends very well to something that we constantly hammer home on my team: using boring tools is often best because it's easier to manage around known problems than forge into the unknown, especially for use-cases that don't have to do with your core business. Extreme & contrived example: it's much better to build your web backend in PHP over Rust because you're standing on the shoulders of decades of prior work, although people will definitely make fun of you at your next webdev meetup.
(Functionality that is core to your business is where you should differentiate and push the boundaries on interesting technology e.g. Search for Google, streaming infrastructure for Netflix. All bets are off here and this is where to reinvent the wheel if you must – this is where you earn your paycheck!)
Thank you for sharing this. I recently started working on metrics at a FAANG and saw some of the challenges you mentioned... the fact that you were able to get good results so quickly is super inspiring!
What's the standard for metrics gathering, push or pull? I prefer pull, but depending on the app it can mean you need to build in a micro HTTP server so there's something to query. That can be a PITA, but pushing a stat on every event seems wasteful, especially if there's a hot path in the code.
I don't think there's any clear standard. There's many confusions about push vs pull that make the discussions hard to follow, as they often make apples to oranges comparisons. For example the push you're talking about in your comment is events, whereas a fair comparison for Prometheus would be with pushing metrics to Graphite. https://www.robustperception.io/which-kind-of-push-events-or... covers this in more detail.
Taking your example you could push without sending a packet on every event by instead accumulating a counter in memory, and pushing out the current total every N seconds to your preferred push-based monitoring system. You could even do this on top of a Prometheus client library, some of the official ones even as a demo allow pushing to Graphite with just two lines of code: https://github.com/prometheus/client_python#graphite
In my personal opinion, pull is overall better than push but only very slightly. Each have their own problems you'll hit as you scale, but those problems can be engineered around in both cases.
There have been a lot of articles posted recently about the 'old' web, and while I like the concept I still have a hard time finding quality information in many of the directories and webrings posted. The level of research and density of information in this blog is very good.
I'm not sure I understood the solution there. Storing only 0.1%-0.01% of the interesting metrics data makes sense in the same way that you'd take a poll of a small fraction of the population to make guesses about the whole?
I believe he means they are storing all the data, but for a subset of types of data, sorta like extracting just a few columns out of a big table. Presumably someone on some other team gets use out of having access to the 99.9% of data stored that is not relevant to "performance or capacity related queries."
[+] [-] bitcharmer|5 years ago|reply
I've done this in fintech a few times already and the best stack that worked from my experience was telegraf + influxdb + grafana. There are many things you can get wrong with metrics collection (what you collect, how, units, how the data is stored, aggregated and eventually presented) and I learned about most of them the hard way.
However, when done right and covering all layers of your software execution stack this can be a game changer, both in terms of capacity planing/picking low hanging perf fruit and day to day operations.
Highly recommend giving it a try as the tools are free, mature and cover a wide spectrum of platforms and services.
[+] [-] chucky_z|5 years ago|reply
I'm in the middle of doing a test-run of vmagent + victoria-metrics as a mostly-replacement (see: not replacing the cases where it's known exactly what's wanted, and used explicitly for that), and victoria-metrics fully support telegraf pushes which is a big bonus.
Also they're apparently going to be dropping InfluxQL entirely, for Flux, which is weird to me? The docs don't outright state this, but InfluxDB 2.0 has only Flux examples in the docs I can find. It's a better language, however the learning curve is not trivial.
[+] [-] benraskin92|5 years ago|reply
[+] [-] chrisweekly|5 years ago|reply
[+] [-] duckmysick|5 years ago|reply
Can you share the important things that are overlooked?
[+] [-] jamessun|5 years ago|reply
[+] [-] Simulacra|5 years ago|reply
[+] [-] m463|5 years ago|reply
[+] [-] dirtydroog|5 years ago|reply
[+] [-] resu_nimda|5 years ago|reply
[+] [-] eshyong|5 years ago|reply
[+] [-] brmgb|5 years ago|reply
"I did it by myself in one day, well actually it was one week but had I known the stack I would likely have done it in one day. Oh, and by the way, after that week, there was yet another month of work involving at least two other persons from my team and then even more work from other teams. But let's not dwell on boring details".
It's nearly as infuriating as the "Appendix: stuff I screwed up" which doesn't contain actual screw up. It's a shame because the rest of the writing is interesting and doesn't need to be propped up.
[+] [-] caiobegotti|5 years ago|reply
[+] [-] derivativethrow|5 years ago|reply
Were you able to move past this to read the rest of the article? Because it's a very good article.
[+] [-] waheoo|5 years ago|reply
The post about salary reads much better so might just be an experience thing.
https://youtu.be/vtIzMaLkCaM
[+] [-] roskilli|5 years ago|reply
Anyone interested in this topic might want to check out an issue thread on the Thanos GitHub project. I would love to see M3, Thanos, Cortex and other Prometheus long term storage solutions all be able to benefit from a project in this space that could dynamically pull back data from any form of Prometheus long term storage using the Prometheus Remote Read protocol: https://github.com/thanos-io/thanos/issues/2682
Spark and Presto both support predicate push down to a data layer, which can be a Prometheus long term metrics store, and are able to perform queries on arbitrary sets of data.
Spark is also super useful for ETLing data into a warehouse (such as HDFS or other backends, i.e. see the BigQuery connector for Spark[1] that could write a query from say a Prometheus long term store metrics and export it into BigQuery for further querying).
[1]: https://cloud.google.com/dataproc/docs/tutorials/bigquery-co...
[+] [-] tixocloud|5 years ago|reply
[+] [-] gigatexal|5 years ago|reply
[+] [-] julianeon|5 years ago|reply
I might be a great telecomm tech, a genius even, but once I'm out of a job, I can't build my own telecomm system - that would cost billions. I have to go back to some other telecomm system to start making money again.
But, at a startup, a kicked-out senior engineer can actually pretty much exactly recreate the company; they can do the equivalent of a laid-off telecomm employee starting a new, almost-as-good (except for branding) telecomm company.
No billions in infrastructure required: within a month or two, the cloned company could be near-indistinguishable from the original.
So companies have to pay employees more like partners, instead of employees, because either they pay them as equals or they'll be forced to compete against them, as equal rivals.
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] sprt|5 years ago|reply
[+] [-] renewiltord|5 years ago|reply
Title didn't live up to article imho. But I get it. Thanks for sharing your methods.
[+] [-] simonw|5 years ago|reply
[+] [-] m463|5 years ago|reply
and anyway:
https://wondermark.com/c/2007-10-11-344ennui.gif
[+] [-] heliodor|5 years ago|reply
The missing piece remains to be able to query one system with the query language of the others. For example, query Prometheus using Graphite's query language.
[+] [-] sa46|5 years ago|reply
We killed our influx cluster numerous times with high cardinality metrics. We migrated to Datadog which charges based on cardinality so we actively avoid useful tags that have too much cardinality. I’m investigating Timescale since our data isn’t that big and btrees are unaffected by cardinality.
[+] [-] RabbitmqGuy|5 years ago|reply
[+] [-] staysaasy|5 years ago|reply
It extends very well to something that we constantly hammer home on my team: using boring tools is often best because it's easier to manage around known problems than forge into the unknown, especially for use-cases that don't have to do with your core business. Extreme & contrived example: it's much better to build your web backend in PHP over Rust because you're standing on the shoulders of decades of prior work, although people will definitely make fun of you at your next webdev meetup.
(Functionality that is core to your business is where you should differentiate and push the boundaries on interesting technology e.g. Search for Google, streaming infrastructure for Netflix. All bets are off here and this is where to reinvent the wheel if you must – this is where you earn your paycheck!)
[+] [-] mv4|5 years ago|reply
[+] [-] tixocloud|5 years ago|reply
[+] [-] chrchang523|5 years ago|reply
[+] [-] dirtydroog|5 years ago|reply
[+] [-] bbrazil|5 years ago|reply
Taking your example you could push without sending a packet on every event by instead accumulating a counter in memory, and pushing out the current total every N seconds to your preferred push-based monitoring system. You could even do this on top of a Prometheus client library, some of the official ones even as a demo allow pushing to Graphite with just two lines of code: https://github.com/prometheus/client_python#graphite
In my personal opinion, pull is overall better than push but only very slightly. Each have their own problems you'll hit as you scale, but those problems can be engineered around in both cases.
Disclaimer: Prometheus developer
[+] [-] halbritt|5 years ago|reply
[+] [-] chris_f|5 years ago|reply
[+] [-] neoplatonian|5 years ago|reply
Or any good resource which discusses possible optimizations in the infra stack at a more theoretical, abstract, generalizable level?
[+] [-] dmos62|5 years ago|reply
[+] [-] yellowstuff|5 years ago|reply
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] wwarner|5 years ago|reply
[+] [-] Aperocky|5 years ago|reply
[+] [-] m0zg|5 years ago|reply
[+] [-] dandare|5 years ago|reply
I feel like I have an inception. Should "boring, descriptive, names" be the default in all IT?
[+] [-] ertian|5 years ago|reply