I've been working exclusively in implementing BI solutions for the past five years. The thing that depresses me the most is not that BI solutions take forever to implement and cost a lot, but that clients many times just don't understand the data they are trying to report on. Many times it leads to over-engineered BI solutions just for one report that a client says is mission critical, but is never used.
I'm sure more technology focused companies don't have any issues using these self-service models, but you wouldn't believe the innumeracy that some people have in industry.
> one report that a client says is mission critical, but is never used.
My dad started developing software in the late 60s. As a kid (let's say circa 1982, definitely in the minicomputer era), I remember him talking about a problem at work: to do all their daily processing, they needed about 28 hours. A lot of the workload was reporting, so he asked managers what reports were no longer useful. Naturally, he was assured that every report was absolutely vital to proper functioning.
His solution was just to start dropping reports. If anybody complained, he'd put them back in the job list. A significant number of reports went unlamented, and soon the computer was able to complete its daily workload handily.
The lesson I took from this is that expressed desire is often very different than actual need, so separating the two can pay big dividends. I've never used that trick, but the lesson runs all through my methods.
There is an interesting phenomenon that I've noticed about BI: By the time data is appropriately gathered, cleaned, aggregated, and presented, the data is so closely aligned to the decision process that the decisions themselves might as well be automated and optimized.
But that never happens. We build reports so that executives can look at them and feel important while they make slower and less optimal decisions than computers could make.
Forget the data. Many companies don't even understand their own processes. I've gone to so many HR departments and asked them how their hiring process works and met blank stares, because there wasn't a single person who knew the entire process from start to finish. Each individual knew bits and pieces, and made assumptions on how the rest of the process worked. Even though my main role is software implementation (enterprise), I spend a ton of time interviewing people and putting those pieces together. It's probably the least favorite part of my job.
Yep. Any exec or manager looking to buy this service as a solution to their reporting needs should sit down and have a serious talk with their analysts or teams who build their reports first.
Innumeracy is why the mantra of asking "Why?" instead of "How?" also applies to BI reporting. A flashy new tool or report isn't going to help as much as having a builder with industry knowledge or experience.
Technology focused companies can have the same issues. In order to build a clear understanding, it requires not just having the data available (the RIGHT data obviously, and clear and extensive documentation on specific definitions and business meanings), but it also requires education within the organization of how one should look at, evaluate, and understand the different pieces of a business.
The good thing though is that there's a tipping point, where once ~1/3 or so of people fully grok how to evaluate and understand the metrics they're looking at, those who do understand start helping (or calling out) those that don't, lifting competence throughout the organization.
I know marketing directors of large retailers who can't figure out what 10% off a figure is, and get confused as to which is gross and which is net. End up explaining VAT at least once a week.
It's cool to be innumerate, though, and if you can't do this stuff yourself there's always a nerd to blame somewhere nearby.
Having worked in some form BI for almost the entirety of my career now, there is not a single, consistent form of BI dashboard that is prevalent across any company. Every solution ends up being unique, because every company has a unique data set-up, stakeholders, definition of metrics, and access needs.
I've worked with Tableau, Domo, Oracle products, you name it. What's the solution that is passed around the most? Excel sheets, because they travel easily and have all-around permissions.
I've been waiting for an out-of-the-box solution that's at least relatively easy to leverage across different organizations, but I haven't seen a painless one yet.
I'm hopeful that Quicksight, while not the be-all end-all solution, provides an example for others to follow, if it does end up being easy to set up and use.
We use Chartio at my company. If you haven't tried it yet, this product is stellar - almost all of our BI needs are handled directly by stakeholders rather than having to go through an engineer, and their support is very helpful and responsive.
I would consider moving off to something like Quicksight if it supported redis. Some of our BI-related data is stored there, and currently to get at it we have an app that proxies data from there to postgres for Chartio's sake.
There are so many BI tools around that it's hard to figure out which solution is going to be best for a particular use case. Creating further confusion, it seems like most "enterprise" BI products aren't explained properly on their websites and are hidden behind "request a demo". It's nearly impossible to evaluate all the possibilities without going crazy.
I want to be able to generate my domain models in some way. Point and click data descriptions are awful. Letting certain people work with raw data is fine, but a lot of users are going to want to work with names that make sense to them. Let me define models with text, just like ORM models.
I want row based security. Let me assign groups to values on certain models. This essentially boils down to hidden filters and required tables.
It should all be web based. I'm not exposing my database directly to customers.
It should definitely not cost 100k a year.
I like the idea of QuickSight, but I can already see that it's not going to work for my needs. But at least they give an upfront description and price. Here's hoping the pricing model drives down the crazy license fees the other vendors are extracting.
I think the big issue with "bi" is that the term is just too broadly applied to be meaningful anymore. Every piece of software with a chart in it somewhere is billed as BI these days and moreover if you ask 10 business users what they want out of their BI system you'll get descriptions of several very different tools which all happen to produce charts. I'm sorry to say I've seen countless businesses deeply disappointed in their BI tools simply because they assumed since "it's a BI tool" it will do X,Y,Z when in reality there are a bunch of different classes of tools to choose from with very different features. The few options that really cover the spectrum are extremely complex and costly to really get value from over the long term. There's a reason why the analyst industry exists - it's just too much damn work to suss out what all these tools actually do on your own.
I make BI tools for a living and I am still struggling to really do a great job classifying all these types of functionality and communicating it well enough to lead customers to the tools they really need.
Hi @jsmeaton, this is Vincent from Holistics (www.holistics.io).
While our website also has a "request a demo" button in our landing page - It's because we have just launched and are looking to validate some of the use-cases that we have built. Will really appreciate if you can contact us and share with us your thoughts.
Will you mind dropping us a note on our website for us to contact you? It will be interesting to get your feedback, and I suspect what we have built (or are building) may meet some of the things you've listed above, though I still need your validation.
A brief introduction about us. We started off as an internal data dashboard for an online video streaming company, serving a specific reporting niche (not a full fledged BI tool) with use-cases different from BI vendors such as Quicksight or the other vendors.
We're trying out Apache Spark with Apache Zeppelin and it's been a pleasure so far. We faced the same problems that everyone else mentioned here -- data is not accessible to people who need it and every datasource requires different tools.
What we like about Apache Spark is that it can take any source and provide the same very fast and programmatic (code reuse!) interface for analysis. Think JSON data dumps from MixPanel, SQL databases, some Excel spreadsheet someone threw together etc.
Apache Zeppelin is a little bit limited in the visualization that comes out of the box, but the benefits of having a shared data language across the company is just such a huge plus. Also, super easy to add data visualization options and hopefully companies will start to contribute these back to the project.
For me, the most interesting part is SPICE. To implement a data warehouse, one creates a star schema (OLAP) database from a regular OLTP database. This involves massive amount of work. It looks like SPICE aims to replace the need for OLAP database and produce similar data directly from OLTP systems. I would love to know more about this engine. I hope Amazon open sources the engine (I highly doubt that they will do it).
There are a few similar technologies out there some of which are open source. Prestodb comes to mind but there's also some other closed systems like powerbi (with some underlying azure distributed data warehouse bits), etc out there.
What's happening in the industry I think is that there was a first wave of data discovery products (tableau being the prime example) that seek to get you right to reporting and data exploration side without necessarily having to slog through all the data warehouse design, and now a second wave like powerbi, etc that are pure saas plays that do something very similar are starting to come out.
Amazon takes risks, and ships innovative products and services every month. Their risk taking is relentless and they take failure in their stride. They "get" how to do push.
No other tech company comes close to their pace. And the key to that is the juicy under the radar micro manager that is Jeff Bezos. If there was an award for best tech CEO. 2015, i'd nominate him in a flash.
I'd say Andrew Jassy is the person you are looking for. Jeff Bezos doesn't give a shit about the day to day strategy of AWS or any other existing Amazon business. He has implicitly said as much in his speeches at Amazon. He really only cares about the next billion dollar business to build within Amazon. Jassy leads AWS, which is the only technologically innovative part of Amazon. The retail side of Amazon is a technological black hole...a few standouts swimming amongst a sea of half-implemented non-solutions to problems that no longer exist but can't be changed because legacy.
One of the most destructive aspects of the mythology of modern tech culture is this ridiculous worship of CEOs, as if they are supermen and the thousands of creative people who actually build the products we enjoy are just the gloves these heroes wear. Stop doing this. You're devaluing the worth of everyone here.
Amazon has been busy, but I still have to go with Satya Nadella. Since he's gotten the job in early 2014, Microsoft has shipped amazing things. Many of his decisions have helped redefine company culture, for the better.
Amazon has shipped a lot, but I think the new Microsoft has a bigger impact. Off the top of my head: open-sourcing .NET, SSH to Windows, Surface product line, Hololens development, Windows 10 (hit a few bumps, but free is HUGE), cross-platform software push, stronger open-source commitment...
If I had a choice between working for Bezos or working for Joel Spolsky at half the pay, it wouldn't even be close. Of course, in real life, Fog Creek probably pays more anyways.
He strikes me as someone that had a couple brilliant fundamental insights, which compose the core of the business. But if one starts thinking everything one shits out is gold on the basis of such insights, one ends up with the Fire phone. However, that sort of thing is not likely to be a true problem for the business for quite a while. They can waste a lot of money and time before it ever becomes a problem.
I'm glad to see they put a little more effort into the product page for this release than seems typical for AWS products. It's much easier on the eyes and, at least for me, much more readable.
A lot of what you are paying for in BI solutions is implementation costs and then to a lesser extent yearly maintenance and support costs. I wonder how they intend to lower implementation costs, since that is kind of lengthy and inherently difficult process.
Agreed re Tableau being a competitor. But I'm not surprised they're not a partner. Last quote we got from Tableau for desktop was $2k. This comes in at around $200/year. That is probably where the "1/10th of the cost of traditional BI solutions" comes from at the top of the home page.
The thing about Tableau is that they're really the industry leader in terms of data exploration workflow. Users love the tool so much that they expect any new data system put in place to work with it so even vendors that are trying to put out their own data exploration ui are partnering up with them to provide the best-of-breed sort of option where they can still sell you the data system, but your c-level who will not even consider using anything but Tableau can still use it.
Now that's a nice acronym! I think Amazon have skilled people in charge of marketing. They make their announcements feel exciting but not too "markety".
On the one hand, you're totally right about the annoucements. On the other hand, the names they give their AWS products are pretty mystifying. AWS Snowball?
And as I understand BI, also for when you want complex graphs that can't be easily modeled in Excel.
With BI solutions you can perform "slices" as they call it, of multi-dimensional data (cubes), and then represent that as graphs that can also be used for drill-down on one or more dimensions of said data.
When you have say 17 dimensions, these solutions are easier to use than using excel to try to do the same.
I have only implemented very simple BI solutions a couple of times, so anyone with more experience can correct me if I'm wrong.
Since 2010, Excel has had a native add-in called Power Pivot which can handle 100M+ row fact tables and provides a full dimensional modeling experience.
"Too big for Excel" is, quite literally, a problem of the last decade.
[+] [-] ececconi|10 years ago|reply
I'm sure more technology focused companies don't have any issues using these self-service models, but you wouldn't believe the innumeracy that some people have in industry.
[+] [-] wpietri|10 years ago|reply
My dad started developing software in the late 60s. As a kid (let's say circa 1982, definitely in the minicomputer era), I remember him talking about a problem at work: to do all their daily processing, they needed about 28 hours. A lot of the workload was reporting, so he asked managers what reports were no longer useful. Naturally, he was assured that every report was absolutely vital to proper functioning.
His solution was just to start dropping reports. If anybody complained, he'd put them back in the job list. A significant number of reports went unlamented, and soon the computer was able to complete its daily workload handily.
The lesson I took from this is that expressed desire is often very different than actual need, so separating the two can pay big dividends. I've never used that trick, but the lesson runs all through my methods.
[+] [-] saosebastiao|10 years ago|reply
But that never happens. We build reports so that executives can look at them and feel important while they make slower and less optimal decisions than computers could make.
[+] [-] enraged_camel|10 years ago|reply
[+] [-] bduerst|10 years ago|reply
Innumeracy is why the mantra of asking "Why?" instead of "How?" also applies to BI reporting. A flashy new tool or report isn't going to help as much as having a builder with industry knowledge or experience.
[+] [-] scuba_man_spiff|10 years ago|reply
The good thing though is that there's a tipping point, where once ~1/3 or so of people fully grok how to evaluate and understand the metrics they're looking at, those who do understand start helping (or calling out) those that don't, lifting competence throughout the organization.
[+] [-] madaxe_again|10 years ago|reply
It's cool to be innumerate, though, and if you can't do this stuff yourself there's always a nerd to blame somewhere nearby.
[+] [-] vkb|10 years ago|reply
I've worked with Tableau, Domo, Oracle products, you name it. What's the solution that is passed around the most? Excel sheets, because they travel easily and have all-around permissions.
I've been waiting for an out-of-the-box solution that's at least relatively easy to leverage across different organizations, but I haven't seen a painless one yet.
I'm hopeful that Quicksight, while not the be-all end-all solution, provides an example for others to follow, if it does end up being easy to set up and use.
[+] [-] m0th87|10 years ago|reply
I would consider moving off to something like Quicksight if it supported redis. Some of our BI-related data is stored there, and currently to get at it we have an app that proxies data from there to postgres for Chartio's sake.
[+] [-] jsmeaton|10 years ago|reply
I want to be able to generate my domain models in some way. Point and click data descriptions are awful. Letting certain people work with raw data is fine, but a lot of users are going to want to work with names that make sense to them. Let me define models with text, just like ORM models.
I want row based security. Let me assign groups to values on certain models. This essentially boils down to hidden filters and required tables.
It should all be web based. I'm not exposing my database directly to customers.
It should definitely not cost 100k a year.
I like the idea of QuickSight, but I can already see that it's not going to work for my needs. But at least they give an upfront description and price. Here's hoping the pricing model drives down the crazy license fees the other vendors are extracting.
[+] [-] baconner|10 years ago|reply
I make BI tools for a living and I am still struggling to really do a great job classifying all these types of functionality and communicating it well enough to lead customers to the tools they really need.
[+] [-] aceregen|10 years ago|reply
While our website also has a "request a demo" button in our landing page - It's because we have just launched and are looking to validate some of the use-cases that we have built. Will really appreciate if you can contact us and share with us your thoughts.
Will you mind dropping us a note on our website for us to contact you? It will be interesting to get your feedback, and I suspect what we have built (or are building) may meet some of the things you've listed above, though I still need your validation.
A brief introduction about us. We started off as an internal data dashboard for an online video streaming company, serving a specific reporting niche (not a full fledged BI tool) with use-cases different from BI vendors such as Quicksight or the other vendors.
[+] [-] mirkoadari|10 years ago|reply
What we like about Apache Spark is that it can take any source and provide the same very fast and programmatic (code reuse!) interface for analysis. Think JSON data dumps from MixPanel, SQL databases, some Excel spreadsheet someone threw together etc.
Apache Zeppelin is a little bit limited in the visualization that comes out of the box, but the benefits of having a shared data language across the company is just such a huge plus. Also, super easy to add data visualization options and hopefully companies will start to contribute these back to the project.
[+] [-] solutionyogi|10 years ago|reply
[+] [-] baconner|10 years ago|reply
What's happening in the industry I think is that there was a first wave of data discovery products (tableau being the prime example) that seek to get you right to reporting and data exploration side without necessarily having to slog through all the data warehouse design, and now a second wave like powerbi, etc that are pure saas plays that do something very similar are starting to come out.
[+] [-] JPKab|10 years ago|reply
[+] [-] brador|10 years ago|reply
Amazon takes risks, and ships innovative products and services every month. Their risk taking is relentless and they take failure in their stride. They "get" how to do push.
No other tech company comes close to their pace. And the key to that is the juicy under the radar micro manager that is Jeff Bezos. If there was an award for best tech CEO. 2015, i'd nominate him in a flash.
[+] [-] saosebastiao|10 years ago|reply
[+] [-] astazangasta|10 years ago|reply
[+] [-] ohitsdom|10 years ago|reply
Amazon has shipped a lot, but I think the new Microsoft has a bigger impact. Off the top of my head: open-sourcing .NET, SSH to Windows, Surface product line, Hololens development, Windows 10 (hit a few bumps, but free is HUGE), cross-platform software push, stronger open-source commitment...
[+] [-] gorena|10 years ago|reply
As long as you're not one of his employees, sure.
If I had a choice between working for Bezos or working for Joel Spolsky at half the pay, it wouldn't even be close. Of course, in real life, Fog Creek probably pays more anyways.
[+] [-] edw519|10 years ago|reply
Their risk taking is also very well managed because they're shipping dog food.
Almost everything that we techies see as innovative is really a byproduct of their primary business of shipping "stuff".
Their risk is reduced because they know what they're pushing has already been outrageously successful for their toughest customer: themselves.
That's a valuable lesson for all would-be startup founders.
[+] [-] jrockway|10 years ago|reply
[+] [-] draw_down|10 years ago|reply
[+] [-] andrewstuart2|10 years ago|reply
[+] [-] BinaryIdiot|10 years ago|reply
Seriously though this product page is a huge improvement over some of their previous releases so kudos to them.
[+] [-] ececconi|10 years ago|reply
[+] [-] ticktocktick|10 years ago|reply
[+] [-] realcul|10 years ago|reply
[+] [-] nostromo|10 years ago|reply
[+] [-] liscovich|10 years ago|reply
[+] [-] mmaunder|10 years ago|reply
[+] [-] sokoloff|10 years ago|reply
[+] [-] baconner|10 years ago|reply
[+] [-] codezero|10 years ago|reply
[+] [-] buremba|10 years ago|reply
[+] [-] JPKab|10 years ago|reply
[+] [-] dsjoerg|10 years ago|reply
[+] [-] tidon12|10 years ago|reply
[+] [-] unknown|10 years ago|reply
[deleted]
[+] [-] simo7|10 years ago|reply
[+] [-] erik-n|10 years ago|reply
Now that's a nice acronym! I think Amazon have skilled people in charge of marketing. They make their announcements feel exciting but not too "markety".
[+] [-] Jgrubb|10 years ago|reply
[+] [-] pjc50|10 years ago|reply
[+] [-] genericpseudo|10 years ago|reply
The Amazon solution here, with the stuff they've launched today is;
a) use Kinesis Firehose to take a stream of log events from your app and dump those into S3 (gives you decent-reliability replicated backups)
b) ETL those using COPY into Amazon Redshift (a column store; very fast full-table scans)
c) point QuickShot at Redshift to draw graphs
so, yeah – but most of the data you want likely doesn't fit in Excel (or rather most of the work is reducing the data in size until it could).
[+] [-] saganus|10 years ago|reply
With BI solutions you can perform "slices" as they call it, of multi-dimensional data (cubes), and then represent that as graphs that can also be used for drill-down on one or more dimensions of said data.
When you have say 17 dimensions, these solutions are easier to use than using excel to try to do the same.
I have only implemented very simple BI solutions a couple of times, so anyone with more experience can correct me if I'm wrong.
[+] [-] ececconi|10 years ago|reply
[+] [-] greggyb|10 years ago|reply
"Too big for Excel" is, quite literally, a problem of the last decade.
Edit: typo small != big
[+] [-] sjg007|10 years ago|reply
[+] [-] unknown|10 years ago|reply
[deleted]