The article was fairly devoid of the necessary details. One of the main ones being: how many events are we talking about? 10 events/sec? 100/sec? 10_000/sec ? And what is the size of these events? How many event emitters are connecting to the Redis server?
With the details, it would be a much more interesting post.
Take a look at logstash [1], it's on GitHub [2]. I think it could replace or integrate with RedisLogHandler.
logstash takes logs from various inputs (syslog, files, Redis, HTTP), filters/normalized the formats into JSON, and outputs various formats (ElasticSearch, Redis, MongoDB, Graylog2). There is a WebUI with search and graphs. It's designed to scale-out and run on multiple machines.
Depends on how much you care about latency, right?
The easy solution is just, y'know, write the log to a file and scp it back to some central place every so often. But then you have to either (a) keep track of how much of a file you've copied, which is a pain; or (b) only grab files that you're no longer actively writing to (as determined by naming scheme or something), but that introduces some latency, depending on how often you rotate.
I'm working on a universal subscriber, however, it currently does a nice job with redis. http://sub-watcher.com It forwards messages back to redis and to syslog. And it has several filtering options.
It provides a common logging interface across platforms.
It's extremely simple (the script that pipes syslog output directly into Redis is probably just a couple lines long).
I have to believe that the people who really seem to like syslog have never worked in organizations that had to deploy things like Splunk or (worse) LogLogic and ArcSight just to make sense of the giant morass of useless text gunk they generate.
Have you noticed how none of the cool kids postprocess http log files anymore?
I agree, it is nothing different than syslog if all you want to do is just dump all logs in one file. But one advantage of this approach in our case was that we wanted to show last 1000 critical logs on a web interface and with logs stored in redis, it was pretty easy to do that. And as redis was part of our stack, it was very easy to hook it up for this specific task.
Syslog is a pile of shit, Ted. It's a relic. You clearly happen to love that relic, and I think you should find a way to place it just-so in a nicely lit alcove in your apartment. The rest of us should move on from it. I don't think less of you for admiring it. I have useless old things on display in my house too.
* Freeform text is a terrible way to track system events.
* Periodically rotated flat files are not a great way to store log information.
* Goofy little UDP messages are not a good way to convey system events
* The syslog PRI field dates back to when we exchanged messages with UUCP.
I could keep going, but since you're just going to reply with "lolwut umad?", I'll leave it at that.
The only time in my life I've ever been envious of syslog was when I had to build an aggregator/event correlator for a bunch of telco equipment that only talked TL1.
Pretty much any other time I've had to work with it, I've wished for something better, so I, for one, am very happy by the thought of people starting to "go out of their way to avoid syslog".
[+] [-] ajays|14 years ago|reply
With the details, it would be a much more interesting post.
[+] [-] antoncohen|14 years ago|reply
logstash takes logs from various inputs (syslog, files, Redis, HTTP), filters/normalized the formats into JSON, and outputs various formats (ElasticSearch, Redis, MongoDB, Graylog2). There is a WebUI with search and graphs. It's designed to scale-out and run on multiple machines.
[1] http://logstash.net/ [2] https://github.com/logstash/logstash
[+] [-] mkelly|14 years ago|reply
The easy solution is just, y'know, write the log to a file and scp it back to some central place every so often. But then you have to either (a) keep track of how much of a file you've copied, which is a pain; or (b) only grab files that you're no longer actively writing to (as determined by naming scheme or something), but that introduces some latency, depending on how often you rotate.
[+] [-] StavrosK|14 years ago|reply
[+] [-] aedocw|14 years ago|reply
[+] [-] unknown|14 years ago|reply
[deleted]
[+] [-] rbucker|14 years ago|reply
[+] [-] andrewvc|14 years ago|reply
[+] [-] tptacek|14 years ago|reply
It's trivially capped and rolled.
It's centralized.
It provides a common logging interface across platforms.
It's extremely simple (the script that pipes syslog output directly into Redis is probably just a couple lines long).
I have to believe that the people who really seem to like syslog have never worked in organizations that had to deploy things like Splunk or (worse) LogLogic and ArcSight just to make sense of the giant morass of useless text gunk they generate.
Have you noticed how none of the cool kids postprocess http log files anymore?
[+] [-] theonlyroot|14 years ago|reply
[+] [-] tedjdziuba|14 years ago|reply
[+] [-] tptacek|14 years ago|reply
* Freeform text is a terrible way to track system events.
* Periodically rotated flat files are not a great way to store log information.
* Goofy little UDP messages are not a good way to convey system events
* The syslog PRI field dates back to when we exchanged messages with UUCP.
I could keep going, but since you're just going to reply with "lolwut umad?", I'll leave it at that.
[+] [-] m0nastic|14 years ago|reply
Pretty much any other time I've had to work with it, I've wished for something better, so I, for one, am very happy by the thought of people starting to "go out of their way to avoid syslog".