It's enterprise software and they are the market leaders for the whole "unstructured data analysis" segment (which is a real segment, and has real demand)
That makes the business essentially a sales organization with a R&D wing. The CEO "is well known for taking Hyperion from $500M in 2001 to revenues of almost $1B in 2007"[1], so he seems to understand that process well.
I also used to have doubt about it because I heard it was categorized as a "log analysis", which is very unsexy from a developer's point of view. Can we be tricked by names! After using it for a while just yesterday, I found it not bad at all.
In a nutshell, Splunk is valuable because it turns your unstructured untyped data into something much easier to analyze and Business Intelligence (BI) ready. IMHO, the strength of Splunk is two folds. First, it provides a central place to view and mine your logs across hundreds or thousands of machines. This is very useful in many organizations with large scale distributed computing because you don't need to ssh into your database server in one terminal and compare the results in another terminal connecting to your web server, if you could ever figure out which of the 100 database servers you should look at. Second, instead of just greping, it treats each log entry as a list of key-value pairs and provides a simple yet powerful query language for it. On top of that, it gives you a library of visualization components. I was able to build a fully-customized Google Analytics style dashboard for my application in 20 minutes. And I would say that is empowerment, and empowerment is the most valuable value.
And if you are very into technologies, they are a Python house, which is always a plus :)
That's mostly because their target customer is big enterprise who feeds on buzzwords they can feed to upper management. It's similar to the websites of SAP, Oracle and Intel, which all have the same qualities:
- No up-front pricing, you need to get a quote
- No screen-shots, you need to see a prepared demo
- Any contact what the company results in assignment of an "account manager", the used-car salesman of the software industry.
- Lots of testimonials from people you've never heard of talking about software you've never heard of using the very same buzzwords found on the website.
That's because they're a log analysis company trying to ride the wave of Big Data hype. It doesn't really fit but if you squint and look at it in the right light you can imagine that it does...
Splunk in a nutshell: it runs a daemon on all your boxes, that tails all your logfiles (you can configure which ones) and pipes them all to a central logging server. On the logging server, another daemon runs that does pattern matching/filtering on the incoming logs, and fires an alert when it gets a match. And umm, that's about it really.
The one useful thing it can do is give (say) devs access to logfiles from prod servers that they aren't allowed for whatever reason to log into themselves. But you could do this yourself with a periodic rsync to an internal webserver...
Splunk is one of the best pieces of software ever made. It's critical to almost everything Shopify does. It's pricy but easy to justify because it makes any ops and many dev tasks 10x easier and therefore there are some real savings in required manpower.
The push away from something very well understood and easily explained (syslog and SIEM aggregation) to calling themselves "big data" and "business analytics" is kind of lame, IMO. I assume they were pushed to do this by the investment market, since big data is hot.
I use Splunk fairly heavily at work with a fairly significant amount of data going through it daily.
It's pretty good software, and isn't as trivial as some on this thread seem to think (I've built large Solr implementations too, so I know search reasonably well).
The strong points: good interface, excellent data import, decent search language, decent docs & community, good APIs, a good set of mostly decent drop in applications that run on top of it.
The weaknesses: While indexeing is Map/Reduced based and scales fairly well, querying is single threaded. That limits it to the performance of a single CPU core + IO limitations. This also applies to things like sub-queries: in a database they could be run separatly, but in Splunk they aren't.
It is also fairly expensive at large scale, although the licening model is fair (it is licensed by data volume, so you can install it on as many machines as you like and share the license between them).
Looking through their site it looks like a Splunk search (or a query) can include a large set of non-trivial, CPU intensive operations. Therefore, perhaps, it is a process that does not lending itself well to multithreading: http://splunk-base.splunk.com/answers/12027/singlemulti-thre...
Congratulations to Splunk! Operational Analytics is an exteremely interesting area rich with many complex problems: performance (processing complex queries over very large data sets), prediction (proactive diagnostics), and visualization (packing dense information in a human-friendly manner).
We at Pattern Insight are currently working on the next generation of logging software, called Log Insight. If you are a Splunk customer or thinking of buying them, take a look at at our product page [http://patterninsight.com/products/log-insight/] for information on a more sophisticated and complete solution.
Don't hesitate to contact me if you are on the job market and want to work on interesting data-mining problems (full-time only please).
Go straight to scale with http://www.sensage.com. Used by governments, health records and insurance companies, telcos, etc. When commodity solutions like Splunk no longer work for you take it to the only other level you need.
Is this type of comment considered OK in this community? I'm all for a little competition, but it feels a bit dirty to me to congratulate a company while calling their product a "[less] sophisticated and complete solution."
Question for those who have used Splunk and alternatives:
How much of what can be done in Splunk, is not possible with alternatives like Graylog2? http://www.graylog2.org/
I considered splunk at some point until I read the EULA review of theirs at http://blog.hacker.dk/
"
Upon at least ten (10) days prior written notice, Splunk may audit your use” …. ” Any such audit will be conducted during regular business hours at your facilities“ … “You will provide Splunk with access to the relevant records and facilities“
"
Why is that them screwing their customers? They sell licenses. Customers agree to use only what they're paying for. This language gives them the right to check and see how many licenses the customers use just to make sure everything's square. Seems fair to me.
The only clause I see worth objecting to is the bit that forbids publishing benchmarks or reviews. That part's BS, but I imagine that it's BS that they would drop rather than lose a sale.
This is entirely par for the course, and quite reasonable if you understand how it happens in practice and why it exists.
There are Splunk customers who do not allow outbound connections to the Internet and so Splunk can't use automated means of auditing license compliance. So they reserve the right to audit you on site. So if you're the CIA and you are paying for 1 petabyte and you are using 2, they want to know and charge you appropriately.
As a matter of good corporate governance, they are actually doing you a favor and preventing you from being a thief. :-)
Is that really screwing their customers? They license by GB/day usage. As expensive as Splunk is, an audit if they think you're really underreporting usage doesn't seem too far fetched.
[+] [-] jahewson|14 years ago|reply
Do they have a viable business or is this just another "inflate and escape" company?
[+] [-] sachinag|14 years ago|reply
[+] [-] nl|14 years ago|reply
It's enterprise software and they are the market leaders for the whole "unstructured data analysis" segment (which is a real segment, and has real demand)
That makes the business essentially a sales organization with a R&D wing. The CEO "is well known for taking Hyperion from $500M in 2001 to revenues of almost $1B in 2007"[1], so he seems to understand that process well.
[1] http://www.splunk.com/view/about-us/SP-CAAAAH8
[+] [-] tocomment|14 years ago|reply
Who designs these things? I gave up after two minutes.
[+] [-] eddieplan9|14 years ago|reply
In a nutshell, Splunk is valuable because it turns your unstructured untyped data into something much easier to analyze and Business Intelligence (BI) ready. IMHO, the strength of Splunk is two folds. First, it provides a central place to view and mine your logs across hundreds or thousands of machines. This is very useful in many organizations with large scale distributed computing because you don't need to ssh into your database server in one terminal and compare the results in another terminal connecting to your web server, if you could ever figure out which of the 100 database servers you should look at. Second, instead of just greping, it treats each log entry as a list of key-value pairs and provides a simple yet powerful query language for it. On top of that, it gives you a library of visualization components. I was able to build a fully-customized Google Analytics style dashboard for my application in 20 minutes. And I would say that is empowerment, and empowerment is the most valuable value.
And if you are very into technologies, they are a Python house, which is always a plus :)
[+] [-] thematt|14 years ago|reply
- No up-front pricing, you need to get a quote
- No screen-shots, you need to see a prepared demo
- Any contact what the company results in assignment of an "account manager", the used-car salesman of the software industry.
- Lots of testimonials from people you've never heard of talking about software you've never heard of using the very same buzzwords found on the website.
[+] [-] jasonkolb|14 years ago|reply
[+] [-] gaius|14 years ago|reply
The one useful thing it can do is give (say) devs access to logfiles from prod servers that they aren't allowed for whatever reason to log into themselves. But you could do this yourself with a periodic rsync to an internal webserver...
[+] [-] xal|14 years ago|reply
[+] [-] rdl|14 years ago|reply
[+] [-] unknown|14 years ago|reply
[deleted]
[+] [-] zackattack|14 years ago|reply
[+] [-] nl|14 years ago|reply
It's pretty good software, and isn't as trivial as some on this thread seem to think (I've built large Solr implementations too, so I know search reasonably well).
The strong points: good interface, excellent data import, decent search language, decent docs & community, good APIs, a good set of mostly decent drop in applications that run on top of it.
The weaknesses: While indexeing is Map/Reduced based and scales fairly well, querying is single threaded. That limits it to the performance of a single CPU core + IO limitations. This also applies to things like sub-queries: in a database they could be run separatly, but in Splunk they aren't.
It is also fairly expensive at large scale, although the licening model is fair (it is licensed by data volume, so you can install it on as many machines as you like and share the license between them).
[+] [-] invertd|14 years ago|reply
[+] [-] jayp|14 years ago|reply
We at Pattern Insight are currently working on the next generation of logging software, called Log Insight. If you are a Splunk customer or thinking of buying them, take a look at at our product page [http://patterninsight.com/products/log-insight/] for information on a more sophisticated and complete solution.
Don't hesitate to contact me if you are on the job market and want to work on interesting data-mining problems (full-time only please).
[+] [-] nwatson|14 years ago|reply
[+] [-] sakai|14 years ago|reply
[+] [-] spot|14 years ago|reply
[+] [-] JumpCrisscross|14 years ago|reply
[1] http://www.sec.gov/Archives/edgar/data/1353283/0001047469120...
[+] [-] ashayh|14 years ago|reply
[+] [-] AdamGibbins|14 years ago|reply
[+] [-] Symmetry|14 years ago|reply
[+] [-] tully|14 years ago|reply
[deleted]
[+] [-] nn2|14 years ago|reply
" Upon at least ten (10) days prior written notice, Splunk may audit your use” …. ” Any such audit will be conducted during regular business hours at your facilities“ … “You will provide Splunk with access to the relevant records and facilities“ "
More or less they screw their customers.
[+] [-] wpietri|14 years ago|reply
The only clause I see worth objecting to is the bit that forbids publishing benchmarks or reviews. That part's BS, but I imagine that it's BS that they would drop rather than lose a sale.
[+] [-] davidu|14 years ago|reply
There are Splunk customers who do not allow outbound connections to the Internet and so Splunk can't use automated means of auditing license compliance. So they reserve the right to audit you on site. So if you're the CIA and you are paying for 1 petabyte and you are using 2, they want to know and charge you appropriately.
As a matter of good corporate governance, they are actually doing you a favor and preventing you from being a thief. :-)
[+] [-] MichaelGG|14 years ago|reply