At any given time, there are a near infinite number of things your business "needs" to do: add a dashboard, A/B test your landing pages, test your database backups, fix your security holes, improve your customer service, add more metrics tracking, etc, etc. The list goes on and on.
A big part of running your business involves looking at this list and picking the ones that make sense for your business, not what the blog-post-du-jour is telling you.
Those are all good things but to me there's only one thing on your list that can instantly end just about any company with no way to recover. That's not having working backups. I would always put that one first!
I'd argue that having a well illustrated overview of your whole company helps you prioritize this list.
So put business and technological metrics on that dashboard. And then you can decide whether you should be focusing most on customer support, A/B testing, or that network bottleneck.
I worked at a place doing network appliance stuff. We managed and developed the machines, and product to end users. A simple dashboard increased our effectiveness by an order of magnitude. Some simple graphs and deep knowledge of the code really really helped. Some data from other departments combined with this was amazing. A simple color coded call queue count told us there was a problem before the front-line tech support people could even report it. Simple analysis of that queue told us where to look for problems: a predominance of one area code told us to examine specific machines (and which subset!), while a wide array of area codes told us that it was probably in billing or comms infrastructure. A couple of graphs of network traffic and system load told me what what the problem could be. I don't know how, I couldn't write code to do it, but I certainly could get to the problem much faster with that minimal input from graphs.
My working theory on dashboards is this: the human mind is amazing at pattern recognition. We're wired for it. We can make intuitive leaps see patterns that may be very hard to describe in math/stats or code. Particularly visual ones. So if you provide that data to your brain to crunch, you are enabling and augmenting your natural tooling. The graphs and stats should be as specific an pre-thought out as possible, but they aren't perfect. Fortunately, as time progresses you learn what "looks right" and what "looks like a problem in subsystem Foo".
This isn't a silver bullet, but certainly it is a great tool. Since then, I've tried to never do work without some sort of visual feedback I can background my innate pattern matching on. Even if it is just scrolling logs -- these patterns emerge and provide clues even if you can't express what they are.
(Anecdote: I had built a demo a while back, and it hiccuped during the live presentation. Fortunately I was in the back of the room with my logs scrolling, and I noticed the logs looked wrong, so I found out a script had died. I restarted it, causing a weird blip in one of our display graphs, but the presenter noticed it before calling attention to that graph in the course of presentation. He glanced at me and I gave him the thumbs up and the audience never even noticed. What was the pattern? The scrolling in one of my log windows slowed down....)
I generally paraphrase this effect as such ;-): "There is no aberration detection mechanism more sophisticated than your marketing guy glancing at a dashboard while drinking his coffee."
I completely agree with your point 'the human mind is amazing at pattern recognition'. From personal experience, I can say I am a 'visual' person. (So much so that my startup is entirely based on this premise, because I think our eyes are amazing analytics tools.) But to my point: is that the case for everybody? Is everyone 'visual', or do some folks think better 'visually' than others?
On one side- yes it's supercool to have a dashboard with maps and metrics and everything like that. And I can see it could be really useful too when something's going wrong.
On the other hand, I'd be wary of building anything like this until I had a good number of customers, paying me money for a service. Is the product so complete (I would ask myself) and my users so happy, that we have no more important task than building a dashboard to look at in the office. Don't some simple email alerts cover most of the important bases (server down, site down etc) for now.
Your startup needs customers (or failing that, users). Your users don't care if you've got a dashboard, they care that your product does what it says.
My sense is that people continue to think that monitoring is hard mostly because it used to be. There are great open source tools and services available now to make both collecting the data and watching it pretty simple to set up.
Yes you need some customers, otherwise you have no data to look at. However, as soon as you have _any_ customers if you aren't looking at real data to decide what is important to do next you are essentially just guessing.
For a long time that's what we did because tracking what was relevant was just too hard. Now what I see in a lot of startups that are leading the pack is that they are monitoring everything whether they think they'll need it or not - then when they have a critical question the data is already there.
Dashboards don't have to be expensive to build. My company uses Graphite now, but before we were at that level, I just had engineering give me credentials for our DBs, and used ODBC + Excel.
Next thing you know, you have a dashboard tracking:
- Real time revenue/revenue projections
- Real time payments funnel metrics (impressions on payments page, conversions, average transaction size)
- Real time tutorial completion rate by hour (aka application health monitoring)
- Other KPIs tracked by the day (revenue, install, etc.)
In fact, there was a period of time when my dashboard was the most effective health monitoring in the company, and detected an issue before engineering or ops.
I can't agree more. First you need a product that works at least for some users, then when you are expanding the user base it's time for dashboards.
Dashboard answers the question: Is my product right now working for my users?
Eventually it can also show trends in terms of resource usage, strange peaks, etc. But the primary function of dashboard is answering the question: "are my users getting the service?"
Additional note: dashboard works wonders for getting us - engineers - understand that there are real users out there right now and thus making us want to build even better product.
I have taken the time to build a graphite/stasd setup. It is over engineered for now, but it grows and is really easy to add in new metrics. (there is even a fab file to install on ubuntu buried in github/lifeisstillgood/frozone)
but, you are right - a dashboard that measures how the site is going is useful. One that measures cashflow, email conversion rates and more is more useful. Linking those into graphite is not a good idea. A more traditional data collection is needed.
Cant agree more. I almost always have email triggers built into the system for ever critical error or failure of the system. When we are being lean on the core product itself building less features... build it when you need it approach ... having a dashboard is kind of luxury.
Metrics are also about the state of your business. What is your growth rate ? What is your churn ? Are people only clicking on certain features ? Where should you improve performance ? These sort of questions are essential for startups especially in the early stages where you need to be careful where to allocate precious resources.
Yes you can sort of gauge that with Google Analytics but only to a certain granularity. Graphite is incredibly flexible.
My new strategy is to instrument the hell out of things -- but only to look at data when enough of it has accumulated to be meaningful.
I think you can make a mistake of measuring and trying to understand data in increments that are so small as to be a waste of time. It can become just another way to not be minding your knitting. Server down? You need to know right now. Knowing that somebody considering buying your product is using a iPhone in New Zealand? Not so much. Computers are great at catching and storing all sorts of stuff. Unless it's critical, best to play with the bells and whistles in controlled deep dives, not as part of a Tony Stark video-game-master-of-the-universe thing. I love bringing up my real-time site metrics and watching all sorts of folks that I am helping. I also like to watch my stock portfolio move throughout the day. But I've found that there's a lot of naval-gazing going on there and not much of anything useful.
Don't look at measurements that generate no subsequent management activity.
We don't really cover any serious metrics, as we're mostly client work - but we did have a big screen that we wanted to use. It's become one of the coolest things in our office.
Something like this would have to be everywhere. I know I always wait a few days, before sending feedback that something doesn't work for me and sometimes I just stop using features like that. The chance to fix something before users would even start to complain is priceless.
Like any doctor would say, the sooner you find out something is wrong, the better. And if nothing goes wrong, you have a huge cool display of how awesome your company is doing.
Dashboards provide value on multiple levels. They radiate information out into your team so everyone is working off of the same shared understanding about the state of the business. They make it easier to detect aberrations. They enable your experts to curate a set of top-level metrics on a per-service basis to look at first when fire-fighting. Every service in your architecture should have it's own dashboard.
If you're interested in dashboards, our startup recently launched a service that makes it simple to hook in your metrics through OSS tools like StatsD and build real-time dashboards with a few clicks: https://metrics.librato.com You just provide the data, we do all the rest. Would love any feedback you might have!
The only issue is the pricing can be a little unfair if you are aiming for more of a real time system since you can't define retention rate. It would be great to be able to model the same system used in Graphite i.e. real time for last 6 hours, then capture every 5 minutes for 6-24 hours and then every 10 minutes from 24 hours onwards.
All of this is well and good, but it's kind of like a "step 1, step ...., PROFIT" joke. The difficult part is picking the right metrics and, often, creating and maintaining them.
It's worth noting that dashboards and instrumentation are one of my key job functions - As it is for all Google SREs. Having the data to tell when you're doing well and when your failing is key - Without knowing you're broke, how do you know to be fixing?
I agree with this. Our version is a plain ol' chalkboard, but it covers the entire wall. We mainly use it for the current state of the company - projects, action steps, projections, etc. Incredibly useful and has a UI everyone understands.
It's not only start-ups that need dashboard or visual metrics, established companies can benefit from more information about how their services/applications/etc work.
Its one thing to give the numbers to the management group, but its more readily accepted when combined in a visual manner. i.e. dashboard
My current employer is only now beginning to see the advantage of collecting good metrics and how it shows the behaviour of both the users and the application.
Their decision making and planning becomes more accurate over time due to having good information at their fingertips rather than an educated guess because originally the system was a black box.
Dashboards are necessary but not sufficient. As your company grows, you also need to send periodic report summaries to people who can act on them. While you certainly need instant pings (emails/IMs/Tweets) in case of major issues (server down, $10k+ shipment got delayed etc.), for the most part, you need to let someone spend 15-30 mins a week to analyze the data and make rational decisions based on that. Dashboards can't help with that but creating too many dashboards will take up valuable resources that could have been used to make more effective reports.
The coolest "hack" about this is that they put the dashboard on several wall-mounted flatscreens.
It doesn't even matter what sort of logging/stats drawing software they use, it could just as well be a tiled window manager with some Google Analytics views on auto-refresh and a colour-coded log tail :)
Not saying you can't do much better than that by picking smart visualisations, but the biggest win is having a few dedicated screens mounted on the wall continuously displaying info, for everyone in the office to glance at.
Not to mention is looks damn cool and impressive :)
GeckoBoard - Dashboard as a service - Can be coerced into creating whatever kind of charts/graphs you want on a slick looking board, but is set to do polling of data which is less helpful for system operations stuff, but nicer for marketing, MAU/DAU monitoring, signups, etc.
I'd also be careful to avoid vanity metrics. I think your time is better spent building a dashboard for actionable metrics. A map of all of your api calls might look cool but what are you going to do with that? I'm more interested in average customer life-time value, churn, customer acquisition cost, retention etc... I've been looking into using geckoboard for this as it seems like it would make setup go much faster.
If you suddenly see a huge cluster of API calls in a single geographic area, that might point to an issue, such as a problem connecting to the local CDN. The stuff you're talking about is actionable from a business standpoint, but the stuff in the article is meant to be actionable from a performance and problem resolution standpoint.
I really, really want to see a beautiful, open source dashboard app, preferably written with Rails.
So far, I've only seen beautiful but closed source SaaS apps, and pretty ugly open source PHP or Python projects.
I'm very tempted to start my own, but I just wish someone like Panic would open source their awesome design.
[+] [-] MicahWedemeyer|14 years ago|reply
A big part of running your business involves looking at this list and picking the ones that make sense for your business, not what the blog-post-du-jour is telling you.
[+] [-] jakejake|14 years ago|reply
[+] [-] drcube|14 years ago|reply
So put business and technological metrics on that dashboard. And then you can decide whether you should be focusing most on customer support, A/B testing, or that network bottleneck.
[+] [-] sophacles|14 years ago|reply
My working theory on dashboards is this: the human mind is amazing at pattern recognition. We're wired for it. We can make intuitive leaps see patterns that may be very hard to describe in math/stats or code. Particularly visual ones. So if you provide that data to your brain to crunch, you are enabling and augmenting your natural tooling. The graphs and stats should be as specific an pre-thought out as possible, but they aren't perfect. Fortunately, as time progresses you learn what "looks right" and what "looks like a problem in subsystem Foo".
This isn't a silver bullet, but certainly it is a great tool. Since then, I've tried to never do work without some sort of visual feedback I can background my innate pattern matching on. Even if it is just scrolling logs -- these patterns emerge and provide clues even if you can't express what they are.
(Anecdote: I had built a demo a while back, and it hiccuped during the live presentation. Fortunately I was in the back of the room with my logs scrolling, and I noticed the logs looked wrong, so I found out a script had died. I restarted it, causing a weird blip in one of our display graphs, but the presenter noticed it before calling attention to that graph in the course of presentation. He glanced at me and I gave him the thumbs up and the audience never even noticed. What was the pattern? The scrolling in one of my log windows slowed down....)
[+] [-] josephruscio|14 years ago|reply
[+] [-] sonofabell|14 years ago|reply
[+] [-] tomgallard|14 years ago|reply
On one side- yes it's supercool to have a dashboard with maps and metrics and everything like that. And I can see it could be really useful too when something's going wrong.
On the other hand, I'd be wary of building anything like this until I had a good number of customers, paying me money for a service. Is the product so complete (I would ask myself) and my users so happy, that we have no more important task than building a dashboard to look at in the office. Don't some simple email alerts cover most of the important bases (server down, site down etc) for now.
Your startup needs customers (or failing that, users). Your users don't care if you've got a dashboard, they care that your product does what it says.
[+] [-] Swizec|14 years ago|reply
Sometimes you need a way to look at the system and say "Hmm, everything works, but those two queues shouldn't have 20k items waiting in them ..."
[+] [-] nextmat|14 years ago|reply
Yes you need some customers, otherwise you have no data to look at. However, as soon as you have _any_ customers if you aren't looking at real data to decide what is important to do next you are essentially just guessing.
For a long time that's what we did because tracking what was relevant was just too hard. Now what I see in a lot of startups that are leading the pack is that they are monitoring everything whether they think they'll need it or not - then when they have a critical question the data is already there.
[+] [-] sardonicbryan|14 years ago|reply
Next thing you know, you have a dashboard tracking: - Real time revenue/revenue projections - Real time payments funnel metrics (impressions on payments page, conversions, average transaction size) - Real time tutorial completion rate by hour (aka application health monitoring) - Other KPIs tracked by the day (revenue, install, etc.)
In fact, there was a period of time when my dashboard was the most effective health monitoring in the company, and detected an issue before engineering or ops.
[+] [-] andraz|14 years ago|reply
Dashboard answers the question: Is my product right now working for my users?
Eventually it can also show trends in terms of resource usage, strange peaks, etc. But the primary function of dashboard is answering the question: "are my users getting the service?"
Additional note: dashboard works wonders for getting us - engineers - understand that there are real users out there right now and thus making us want to build even better product.
[+] [-] lifeisstillgood|14 years ago|reply
I have taken the time to build a graphite/stasd setup. It is over engineered for now, but it grows and is really easy to add in new metrics. (there is even a fab file to install on ubuntu buried in github/lifeisstillgood/frozone)
but, you are right - a dashboard that measures how the site is going is useful. One that measures cashflow, email conversion rates and more is more useful. Linking those into graphite is not a good idea. A more traditional data collection is needed.
[+] [-] Akram|14 years ago|reply
[+] [-] Kiro|14 years ago|reply
[+] [-] taligent|14 years ago|reply
Metrics are also about the state of your business. What is your growth rate ? What is your churn ? Are people only clicking on certain features ? Where should you improve performance ? These sort of questions are essential for startups especially in the early stages where you need to be careful where to allocate precious resources.
Yes you can sort of gauge that with Google Analytics but only to a certain granularity. Graphite is incredibly flexible.
[+] [-] DanielBMarkham|14 years ago|reply
I think you can make a mistake of measuring and trying to understand data in increments that are so small as to be a waste of time. It can become just another way to not be minding your knitting. Server down? You need to know right now. Knowing that somebody considering buying your product is using a iPhone in New Zealand? Not so much. Computers are great at catching and storing all sorts of stuff. Unless it's critical, best to play with the bells and whistles in controlled deep dives, not as part of a Tony Stark video-game-master-of-the-universe thing. I love bringing up my real-time site metrics and watching all sorts of folks that I am helping. I also like to watch my stock portfolio move throughout the day. But I've found that there's a lot of naval-gazing going on there and not much of anything useful.
Don't look at measurements that generate no subsequent management activity.
[+] [-] chucklarge|14 years ago|reply
http://www.perceptualedge.com/articles/Whitepapers/Dashboard... http://www.perceptualedge.com/articles/Whitepapers/Common_Pi...
The author also has written a great book on dashboard design.
Operations dashboards are essential for any (tech?) company.
Business dashboards are another beast all together.
[+] [-] j_s|14 years ago|reply
Do you have any additional suggested reading material? Thanks much.
[+] [-] elliottkember|14 years ago|reply
We don't really cover any serious metrics, as we're mostly client work - but we did have a big screen that we wanted to use. It's become one of the coolest things in our office.
[+] [-] onlyup|14 years ago|reply
[+] [-] pcopley|14 years ago|reply
[+] [-] zidar|14 years ago|reply
Like any doctor would say, the sooner you find out something is wrong, the better. And if nothing goes wrong, you have a huge cool display of how awesome your company is doing.
[+] [-] josephruscio|14 years ago|reply
If you're interested in dashboards, our startup recently launched a service that makes it simple to hook in your metrics through OSS tools like StatsD and build real-time dashboards with a few clicks: https://metrics.librato.com You just provide the data, we do all the rest. Would love any feedback you might have!
[+] [-] taligent|14 years ago|reply
The only issue is the pricing can be a little unfair if you are aiming for more of a real time system since you can't define retention rate. It would be great to be able to model the same system used in Graphite i.e. real time for last 6 hours, then capture every 5 minutes for 6-24 hours and then every 10 minutes from 24 hours onwards.
[+] [-] saint-loup|14 years ago|reply
[+] [-] Swizec|14 years ago|reply
[+] [-] GauntletWizard|14 years ago|reply
[+] [-] tomkin|14 years ago|reply
[+] [-] plant42|14 years ago|reply
Its one thing to give the numbers to the management group, but its more readily accepted when combined in a visual manner. i.e. dashboard
My current employer is only now beginning to see the advantage of collecting good metrics and how it shows the behaviour of both the users and the application.
Their decision making and planning becomes more accurate over time due to having good information at their fingertips rather than an educated guess because originally the system was a black box.
[+] [-] chime|14 years ago|reply
[+] [-] tripzilch|13 years ago|reply
It doesn't even matter what sort of logging/stats drawing software they use, it could just as well be a tiled window manager with some Google Analytics views on auto-refresh and a colour-coded log tail :)
Not saying you can't do much better than that by picking smart visualisations, but the biggest win is having a few dedicated screens mounted on the wall continuously displaying info, for everyone in the office to glance at.
Not to mention is looks damn cool and impressive :)
[+] [-] michaelbuckbee|14 years ago|reply
Stashboard - Open source status monitor for elements of your infrastructure, run it on AppEngine
http://www.stashboard.org/
GeckoBoard - Dashboard as a service - Can be coerced into creating whatever kind of charts/graphs you want on a slick looking board, but is set to do polling of data which is less helpful for system operations stuff, but nicer for marketing, MAU/DAU monitoring, signups, etc.
[+] [-] pmjoyce|14 years ago|reply
Disclosure: I'm one of the founders, thanks for the mention.
[+] [-] 3amOpsGuy|14 years ago|reply
We have plenty of tasks we could constructively spend our time on rather watschen der blinkenlichten.
If no one's going to watch your dashboard, why build it? If occasional glances at a dashboard is your monitoring strategy, why monitor at all?
Our monitoring system has to not only interrupt and alert us, it also has to be capable of some interventions on our behalf occasionally.
[+] [-] BryanB55|14 years ago|reply
[+] [-] mcantor|14 years ago|reply
[+] [-] fennecfoxen|14 years ago|reply
Sprinkle the stat-recording somewhere common to all the calls (metaprogrammy-inheritance-whatever) and it's not even like it's a ton of work.
[+] [-] nathan_f77|14 years ago|reply
So far, I've only seen beautiful but closed source SaaS apps, and pretty ugly open source PHP or Python projects. I'm very tempted to start my own, but I just wish someone like Panic would open source their awesome design.
[+] [-] philip1209|14 years ago|reply
# New Users Today # Returning Users % Conversion Rate in Last 24 Hours % Open Rate on Last Email Campaign $ Dollars Raised Today
[+] [-] kevin1024|14 years ago|reply