Ask HN: Who is using a self hosted analytics system?

[mfrye0]

I'm so used to SASS based solutions these days like Mixpanel, GA, Kiss Metrics, etc. But with everything happening over the last few years in regards to privacy and security I've been looking into self hosted.

Is anyone else considering the same? I know this is standard for companies like Amazon and Facebook, but what about everyone else?

If you are using a self hosted setup, what is it? Custom built, open source, etc?

[mrgreenfur]

If you're serious about self-hosting analytics there is only one serious place to go: http://snowplowanalytics.com/

I don't use them, but they are building enterprise-grade self-hosting.

Disclaimer: I am working on a project in the marketing analytics space. I don't use them and this isn't an endorsement, just a pointer to research more!

[mfrye0]

This looks awesome. Just what I was looking for.

Seems like they have some decent companies using them too.

[ohgh1ieD]

I'm actually on the same road right now, currently I'm testing Piwik.

https://piwik.org

[pesfandiar]

What's your experience with it so far? I was playing around with it at a startup a few years ago. It seemed to have a strong community and features were popping up rather quickly. However, it didn't scale that well for us.

[mfrye0]

Yeah I've been looking at Piwik. What do you think so far?

The only aspect I didn't like was it was PHP. Not really my strong language - if I wanted to customize it.

[gesman]

I built my own analytics App for Splunk to offer business insights for my wife's small business. Mostly how traffic correlates with purchases and where buyers are coming from.

As a side effect same system detects malware and cyber attacks on other websites pretty well as well.

https://splunkbase.splunk.com/app/2676/

[mfrye0]

Whoa. That's badass.

Do you pull any info regarding the IP addresses, or is it only the raw logs that you're going through?

[__d]

I've used https://piwik.org successfully.

[mfrye0]

Thoughts using it so far?

[tixocloud]

To give you some perspective, a startup I worked for began with a self-hosted web analytics option.

In fact, most of the systems we had were all homegrown and while initially it was a great idea - the maintenance part of it took a lot of time away from optimizing on generating revenue.

We spent a lot of time trying to figure out the structure of our self-hosted web analytics platform, how it tracks data, how it stores data, etc. The majority of our company were engineers but we still wasted a lot of time fiddling with the self-hosted analytics.

There will likely be a subset of customers who would be interested in self-hosted but I'm willing to bet that they are more likely to be companies more engineering-oriented.

[mfrye0]

Good feedback.

Yeah I was thinking the same thing - that companies who are self hosting are more likely to be engineering-oriented. Otherwise like you said it's just too much of a pain to handle it yourself.

May I ask, why did you self-host vs use an external service? To save money, for greater security and privacy?

[shakna]

I don't yet have a product I've settled on, but I'm in the early stages of developing a website for a government body.

With all the regulations and policies on data protection, using something not self-hosted is just not going to happen. (I believe that if there was a possibility that their users were affected by a 3rd party breach, the fine is around 1000x the project budget).

If this means a little less information about the audience, that is perfectly acceptable.

[mfrye0]

Yeah that makes sense being a government org.

[boyter]

I have a custom built one. It's main feature is that it can email me the details I want on a scheduled basis to save me logging in.

It tracks a few million records a month so not high scale and runs on a single $5 digital ocean instance.

I did attempt to make it Sass for hosting resellers as an upsell but never made any progress. It's just running for myself these days.

[mfrye0]

Sounds pretty cool. What tech is it based on? Just a standard api and db?

Also, can I ask how big your company is? A few million records is decent volume.

[zyzioziom]

Piwik is pretty good, but like most open source, not so feature packed comparing to commercial tools. It's resource heavy, though. We had problems for big amounts of data which couldn't be loaded before timeouts. Caching helps, but it definitely needs stronger machine.

[mfrye0]

I've been talking to a friend about it and he mentioned using Elastic Search to build our own setup. Idk if it's the right use case though...

[Nilef]

Working for a bank processing millions of calls a minute; using WebTrends

[mfrye0]

I haven't heard of WebTrends before. I'm looking at their website now. Are you self-hosting it? I can't find info on that option.