Anomaly detection: business metrics vs. system metrics?

[chipfixer]

Recently, I ran into someone at a conference who had a major incident: a config change caused a revenue drop. Their RED/system metrics didn’t catch it because they were all static-threshold alerts and siloed from the business signals, so engineers didn't discover the actual revenue impact until much later.

What may have been helpful is anomaly detection directly on their business metrics — with system metrics helping explain root cause but only when real customer/business impact is detected.

Curious to hear: How much does your org prioritize monitoring business metrics (not just System metrics)? If you do, what tools do you use?

[nchinmay]

We had this very issue where a bad configuration change (human error) caused a large & sudden revenue drop and a drop in our streaming ad event metrics. This is a realtime adtech system where a delay in detecting sudden changes in business metrics can have monetary impact and visible customer experience drops. In this case, the major drop in revenue was immediately found and addressed but not all of our expected alerts went off. Our streaming ad events metric threshold was statically set too low. This threshold was appropriate at earlier stages of our business but as our business has grown, this threshold happens to be too low to set off the alert I would have expected to go off as the very first one. We do have sophisticated metrics instrumentation and alerting but an effective anomaly detection around sudden upticks/downticks in business metrics while being conscious of underlying metric trends evolving organically with the business would be a game changer.

Larger, incident-worthy changes in metrics are also easier to set static thresholds around and ring more than one bell when they occur. I'd be more concerned about smaller to mid deviations from the trend, say, sudden -/+10% change in my business metrics over X minutes. Can I reliably set a static threshold that will universally be appropriate here? A good anomaly detector would ideally bring something like this to attention without hard coded alert configs here

[poobear22]

I managed the system administrators for a high performance computing center. We took a lot of blame for the applications when in reality, often times it was poor programming on the developer's part. So, I got really tired of taking the blame and implemented statistical process control to track the mean time between failures of the jobs. I was really just shining the flashlight on production jobs and was hoping it could change the culture. It was not my job to fix their code, and the applications were developed by a different group of people with a very different culture. I thought the process control worked really well, and it did allow me to take the heat off me for random blaming of my team, when I could respond with "your job is failing XX times per year" and from there, push for a root cause analysis. But pushing against that culture was really hard, and there was a lot of "set the job to complete and I will look at it on Monday". If they do not want to conduct a root cause analysis on the failure modes for their code, I can't do much. So, even implementing some type of monitoring can have little effect if the ones who need to fix something do not support the culture. And, as I read your post, I'd think people would be looking at these business metrics a little closer or develop more sensitive metrics to catch these issues.

[chipfixer]

Yup, no amount or type of anomaly detection can fix the culture. That said, in this case, maybe one reason it may be hard is the devs weren't the ones owning what the job did in production?