One of the funny things is that I've still never really used Redis directly, only as a sort of opaque part of other things.
I posted that link because I've known antirez for a long time, worked with him at Linuxcare, and when he said he was working on something cool - I believed him!
In 2011 I wrote PHP client for Redis with features like tags, safe locks, map/reduce. It was heavily tested and users had no issues - and since there was no issues, it was removed from the list of clients after few years as "abandoned". Nobody even tried to contact me :)
Now I don't use PHP and barely use Redis so it's all in past. But that experience taught me a lot of new things.
Salvatore Sanfilippo, you are great programmer. Redis API is elegant, safe and effective. I personally had 0 data losses all years I use Redis. And this DB is always extremely fast.
That is the irony of software development. In theory, good software could last forever without any maintenance. In practice though, nothing has to be replaced as often as software ;-)
What does that teach us about our software development practices?
My most popular open source library is a Redis backed cache for Ruby. It continues to accumulate stars, but I haven’t had any issues in well over a year.
Sometimes software is just finished, not abandoned!
One thing I think OSS developers / projects would benefit from is (self) certification.
Let's say you write a great piece of software that has no need of updating. But Redis keeps evolving - making new releases.
You could just each release of Redis run a series of checks and add a "verification" that we support version 1.0,1.1,1.2 of redis.
But even better is if someone else does this - why? Because it solves a problem I have seen a lot on government circles - the "we cannot use OSS because it is not supported"
But if another company says "we have reviewed, tested and used" version X of php-Redis then you suddenly have a self supporting eco-system.
Webshops that care about winning tenders can say "this software is supported by dozens of providers around the globe" so if it's original user goes off line there are still people who provably have skills and experience with it.
Everyone wins
(note - I am in no way saying this is something you should have done or thought about or arranged - writing software is hard enough without these sort of long term, unprofitable, activities - it's just matches observations i have on getting oss into government (plug: http://www.oss4gov.org/manifesto)
That must have been annoying. It’s interesting, I’ve been evaluating react css frameworks these last couple of days and one of things I’m looking at it is commit log to see what activity is like on the project. It did cross my mind that ones that had gone quiet might be very complete and bug free...but I decided that would be the exception.
Anyway, kudos for writing the client and kudos to Redis. It’s a brilliant bit of software.
For me, Redis is the most software valuable tool I use.
I was hooked from the day I started using it about 8 years ago. It somehow has a place in almost every software project
I work on. And it just keeps getting better.
Now with RESP3 on the horizon, it will enable me to even
further benefit from the many clients available in almost
every programming language.
I can truly say that Redis has changed my life and career. I started using it in 2010, got hooked immediately, and I've been a member of the community and contributor ever since. I got to build so many cool things with Redis over the years, it's amazing how versatile and powerful it is.
Having worked for a couple of years at Redis Labs, I got to work closely with antirez, and that's been a transformative experience as well, which made me a better engineer and open source contributor.
Thank you, Salvatore. Here's to the next 10 years.
What's the clustering story these days? Last I checked, Redis Cluster still had lots of issues losing writes, and the design doesn't seem to have been revised since Aphyr ran Jepsen tests on it back in 2013, making it (at least from my perspective) practically useless for anything that requires distributed consistency, which means most things that I look to use a distributed data store for.
I've also been warned by fellow devs that much of the clustering logic is actually done in the client, and there's historically been a lack of mature clients for all languages. Even if you find a mature client, the complexity of the implementation implies that not all clients may behave identically, or may have different bugs. Then there's the issue of lock-in, where you become dependent on a specific client library and its development lifecycle. I don't know if all of this is true, but I also don't hear a lot of people talking about Redis Cluster these days.
I know you can use Redis in master/slave mode via Sentinel + Twemproxy, though even this solution seems to have some issues with data consistency. Running all three also appears a lot more complex than an integrated system.
I see a lot of comments implying that Redis is mainly used in single-node setups, so that might be where it shines?
Redis is the best data store I've had the fortune to work with. No nonsense data structures coupled with extremely reliable performance, beautiful API and extremely easy to manage.
It also helped me earn a lot of praise for using redis-cli --pipe to bulk insert data which bringed down a data import job to a few minutes from 4 hours. I eventually built a wrapper around redis-cli --pipe for using it with a cluster.
Redis is incredible. I built a matchmaker with it as the sole data store and it has been a breeze to use. It's so simple to visualize data structures with it and I just can't see myself using any other in-memory data store. Also Redis Sentinel is fantastic.
I expect it to always work and it always has. I really like it but am worried I trust it too much now. Please tell me I'm fine to trust single instances!
It's been one of the least problematic things in our infrastructure. We keep around for a bit of internal caching of things, transient state, and some non critical queueing. We have a couple of redis nodes that we've had for years. Saying it is a key value store is selling it short. What really makes it useful is things like queues, sets, ttls on keys, etc. The API has dozens of different operations and variants of operations. Mostly, redis is rock solid and stable but because it is not a transactional datastore you should not rely on it preserving your data. Bearing that in mind, you have to plan for the worst. IMHO treating it as a transient thing that can go at any point and that you don't back up is a sensible thing to do. Blindly restarting and wiping a redis node should not cause any harm. Mostly this never happens but when it does, we simply restart.
Redis cluster is more about availability than it is about consistency. If you are aware of that, it's a fine solution.
A couple of things I do with it:
- buffer log messages before we send them to elasticsearch via logstash. This is a simple queue. Technically it's a single point of failure but worst case we lose a few minutes of logging. This happens very rarely and typically not because of redis. This node is configured to drop older keys when memory runs out. We did this after a few log spikes killed our node by running out of memory. Since then, we've had zero issues.
- we have a few simple batch jobs that we trigger with an API or via a timed job in our cluster. To prevent these things running twice on different nodes, I implemented a simple lock mechanism via redis. Nodes that are about to run check first if they need to and abort if another node is already doing the same or recently completed doing this. This does not scale but it works great and I don't need extra moving parts for some routine things that we run a couple of times a day.
- some of our business logic ends up looking up or calculating the same things over and over again. We use a mix of on server in memory caching and shared state in redis for this. Keys for this have a ttl; if the key is missing the logic is to simply recalculate the value.
Once you have redis around, finding more uses for it is a bit of an antipattern. It does queuing but you probably should use a proper queue if you need one. It can store data but you probably want a proper database if you are going to store a lot of data, etc. It's great for prototyping though. Use it in moderation.
Only problem I ever encountered was mainly my fault -- my Redis instance was used to hack my server (the attacker manipulated Redis data and dumped it to overwrite /etc/passwd, etc). I was an idiot and hadn't locked down my installation. Luckily my provider had disk snapshots.
Pretty much the only problems I've seen it cause are due to people not understanding its role in the infrastructure, not defects in Redis itself.
Most people run Redis in-memory only (in my experience, at least). Those that don't usually sync to disk only periodically, whether they intend to or not.
The only problems that crop up from that pattern are that many users (especially new ones, or people who haven't worked with Redis before) forget that it's fundamentally an ephemeral cache. Eventually maintenance or failure drops the in-memory dataset, and then a wide variety of disasters occur because it was being treated as a source of truth, or as a datastore with durability.
In situations where the ephemerality of in-memory data was consistently known (or when disk persistence was configured with some thought), I have had the same experience as most others here: Redis was one of the most reliable, least surprising pieces of infrastructure present.
...except for TTL handling with read-only replicas, I guess. That behavior (TTLs can get ignored on replicas) was really rough and surprising, but is fortunately now fixed. Shame on me for running an old enough version to keep getting bitten by it.
I did, but I am not sure the use case was right. Inherited an old web forms project and tried switching from Memcached to Redis. It didn't work out due to a large number of serialization/deserialization of object stores. The .net Redis library was causing a massive amount of garbage collection storms. I was probably asking it to do too much though.
I was first exposed to Redis when I first started building NodeJS sites using ExpressJS many years ago. There was a session caching layer that used Redis that “just worked” and was fast. Later I had other opportunities to use Redis and the more I used it the more my appreciation grew. However I echo some of the well put discussions in this thread about Redis clustering. Thank you Antirez!
I like antirez's "Linenoise" library. I converted it to Unicode, and gave it a multi-line editing mode with undo, visual copy and paste, edit in external editor, parenthesis matching, incomplete syntax continuation, and more. It's all in the TXR Lisp REPL.
Tcl was the first language used. I laughed, but only because I've developed in Tcl and it has its time and place. Though I don't remember how to do an if statement in Tcl off the top of my head. Update: ok, its squiggly brackets; that's right.
Just coming here to say congratulations to Antirez.
I've never had to use the software myself, but I always read Antirez's posts and his interactions on here. I think it's a great example of technical leadership.
[+] [-] davidw|7 years ago|reply
I posted that link because I've known antirez for a long time, worked with him at Linuxcare, and when he said he was working on something cool - I believed him!
[+] [-] antirez|7 years ago|reply
[+] [-] turdnagel|7 years ago|reply
[+] [-] schoen|7 years ago|reply
[+] [-] EugeneOZ|7 years ago|reply
In 2011 I wrote PHP client for Redis with features like tags, safe locks, map/reduce. It was heavily tested and users had no issues - and since there was no issues, it was removed from the list of clients after few years as "abandoned". Nobody even tried to contact me :) Now I don't use PHP and barely use Redis so it's all in past. But that experience taught me a lot of new things.
Salvatore Sanfilippo, you are great programmer. Redis API is elegant, safe and effective. I personally had 0 data losses all years I use Redis. And this DB is always extremely fast.
[+] [-] akuji1993|7 years ago|reply
[+] [-] arendtio|7 years ago|reply
What does that teach us about our software development practices?
[+] [-] pselbert|7 years ago|reply
Sometimes software is just finished, not abandoned!
[+] [-] lifeisstillgood|7 years ago|reply
Let's say you write a great piece of software that has no need of updating. But Redis keeps evolving - making new releases.
You could just each release of Redis run a series of checks and add a "verification" that we support version 1.0,1.1,1.2 of redis.
But even better is if someone else does this - why? Because it solves a problem I have seen a lot on government circles - the "we cannot use OSS because it is not supported"
But if another company says "we have reviewed, tested and used" version X of php-Redis then you suddenly have a self supporting eco-system.
Webshops that care about winning tenders can say "this software is supported by dozens of providers around the globe" so if it's original user goes off line there are still people who provably have skills and experience with it.
Everyone wins
(note - I am in no way saying this is something you should have done or thought about or arranged - writing software is hard enough without these sort of long term, unprofitable, activities - it's just matches observations i have on getting oss into government (plug: http://www.oss4gov.org/manifesto)
[+] [-] aidos|7 years ago|reply
Anyway, kudos for writing the client and kudos to Redis. It’s a brilliant bit of software.
[+] [-] hartator|7 years ago|reply
[+] [-] azhenley|7 years ago|reply
https://news.ycombinator.com/item?id=494649
[+] [-] bilifuduo|7 years ago|reply
[+] [-] craz8|7 years ago|reply
Ezra was one of the good guys
[+] [-] fyfy18|7 years ago|reply
https://news.ycombinator.com/item?id=8863
[+] [-] misiti3780|7 years ago|reply
[+] [-] lemoncucumber|7 years ago|reply
[+] [-] michaelangerman|7 years ago|reply
https://news.ycombinator.com/item?id=494649
Here is the link from Feb 25, 2009
Note this number = 494649
Today = 19 247493
Nineteen (19) Million Posts etc later (mas o menos)
[+] [-] dvirsky|7 years ago|reply
Having worked for a couple of years at Redis Labs, I got to work closely with antirez, and that's been a transformative experience as well, which made me a better engineer and open source contributor.
Thank you, Salvatore. Here's to the next 10 years.
[+] [-] atombender|7 years ago|reply
I've also been warned by fellow devs that much of the clustering logic is actually done in the client, and there's historically been a lack of mature clients for all languages. Even if you find a mature client, the complexity of the implementation implies that not all clients may behave identically, or may have different bugs. Then there's the issue of lock-in, where you become dependent on a specific client library and its development lifecycle. I don't know if all of this is true, but I also don't hear a lot of people talking about Redis Cluster these days.
I know you can use Redis in master/slave mode via Sentinel + Twemproxy, though even this solution seems to have some issues with data consistency. Running all three also appears a lot more complex than an integrated system.
I see a lot of comments implying that Redis is mainly used in single-node setups, so that might be where it shines?
[+] [-] rkeene2|7 years ago|reply
[+] [-] hashhar|7 years ago|reply
It also helped me earn a lot of praise for using redis-cli --pipe to bulk insert data which bringed down a data import job to a few minutes from 4 hours. I eventually built a wrapper around redis-cli --pipe for using it with a cluster.
[+] [-] techie128|7 years ago|reply
[+] [-] phowon|7 years ago|reply
[+] [-] dimgl|7 years ago|reply
[+] [-] kortilla|7 years ago|reply
[+] [-] MuffinFlavored|7 years ago|reply
[+] [-] whalesalad|7 years ago|reply
[+] [-] incomplete|7 years ago|reply
[+] [-] simonswain|7 years ago|reply
[+] [-] rb808|7 years ago|reply
I expect it to always work and it always has. I really like it but am worried I trust it too much now. Please tell me I'm fine to trust single instances!
[+] [-] jillesvangurp|7 years ago|reply
Redis cluster is more about availability than it is about consistency. If you are aware of that, it's a fine solution.
A couple of things I do with it:
- buffer log messages before we send them to elasticsearch via logstash. This is a simple queue. Technically it's a single point of failure but worst case we lose a few minutes of logging. This happens very rarely and typically not because of redis. This node is configured to drop older keys when memory runs out. We did this after a few log spikes killed our node by running out of memory. Since then, we've had zero issues.
- we have a few simple batch jobs that we trigger with an API or via a timed job in our cluster. To prevent these things running twice on different nodes, I implemented a simple lock mechanism via redis. Nodes that are about to run check first if they need to and abort if another node is already doing the same or recently completed doing this. This does not scale but it works great and I don't need extra moving parts for some routine things that we run a couple of times a day.
- some of our business logic ends up looking up or calculating the same things over and over again. We use a mix of on server in memory caching and shared state in redis for this. Keys for this have a ttl; if the key is missing the logic is to simply recalculate the value.
Once you have redis around, finding more uses for it is a bit of an antipattern. It does queuing but you probably should use a proper queue if you need one. It can store data but you probably want a proper database if you are going to store a lot of data, etc. It's great for prototyping though. Use it in moderation.
[+] [-] Gigablah|7 years ago|reply
[+] [-] zbentley|7 years ago|reply
Most people run Redis in-memory only (in my experience, at least). Those that don't usually sync to disk only periodically, whether they intend to or not.
The only problems that crop up from that pattern are that many users (especially new ones, or people who haven't worked with Redis before) forget that it's fundamentally an ephemeral cache. Eventually maintenance or failure drops the in-memory dataset, and then a wide variety of disasters occur because it was being treated as a source of truth, or as a datastore with durability.
In situations where the ephemerality of in-memory data was consistently known (or when disk persistence was configured with some thought), I have had the same experience as most others here: Redis was one of the most reliable, least surprising pieces of infrastructure present.
...except for TTL handling with read-only replicas, I guess. That behavior (TTLs can get ignored on replicas) was really rough and surprising, but is fortunately now fixed. Shame on me for running an old enough version to keep getting bitten by it.
[+] [-] starik36|7 years ago|reply
[+] [-] dstroot|7 years ago|reply
[+] [-] kazinator|7 years ago|reply
[+] [-] antirez|7 years ago|reply
[+] [-] iokevins|7 years ago|reply
[+] [-] fuball63|7 years ago|reply
[+] [-] yeskia|7 years ago|reply
[+] [-] wnevets|7 years ago|reply
[+] [-] seibelj|7 years ago|reply
[+] [-] tom_|7 years ago|reply
[+] [-] dominotw|7 years ago|reply
[+] [-] gadders|7 years ago|reply
I've never had to use the software myself, but I always read Antirez's posts and his interactions on here. I think it's a great example of technical leadership.
[+] [-] saberience|7 years ago|reply
[+] [-] therealdrag0|7 years ago|reply
[+] [-] kawsper|7 years ago|reply
[+] [-] jaggederest|7 years ago|reply