In-memory database Redis wants to dabble in disk

[mrinterweb]

I was confused because I know Redis already offers persistence. https://redis.io/docs/management/persistence/ The main difference here is how its runtime will shift from runtime data only in RAM to not storing as much data in RAM and looking up on disk. Current Redis persistence looks like the runtime is stored in RAM and changes are periodically or append only persisted to disk. I hope I got that right.

[hivacruz]

For those interested in an alternative to Redis on disk, compatible with the Redis protocol, have a look on Kvrocks[1] which was recently accepted in the Apache foundation. It is based on RocksDB and works quite nicely for us.

[1]: https://github.com/apache/kvrocks

[chocolatkey]

Thanks for this, it seems to even support redis' lua functionality with is essential for us. I wish I hadn't decided to use redis stack since even kvrocks isn't compatible with its extensions

[donatj]

> It is part of the drive to make Redis "more like your classic database," he said. In the future, support for natural language queries and enhanced vector and feature store capabilities will be added. This initiative aligns with Redis's ambition to be seen as more than just a fast, albeit expensive, cache

This sounds to me like not understanding your place in the market or why people use your product. I can name half a dozen classic SQL databases off hand. I can’t name a tool that competes with Redis at their niche. Why aim for an already oversaturated market when you already have a good profitable niche.

[vidarh]

> Why aim for an already oversaturated market when you already have a good profitable niche.

Presumably because there are a lot of things that straddles the two, and where you risk losing people to databases or hybrid setups where the Redis use will be simple enough that people won't want to pay for support.

E.g. cases where you can't afford to keep it all in memory where you today might end up resorting to using Redis as just a dumb-ish in front of a database, or even avoiding Redis entirely in favour of in-memory tables or aggressive cache settings for your database even in cases where you might prefer Redis if it could intelligently handle a larger working set more cheaply.

And also to create a buffer given that the level of oversaturation in the classic database market may make their niche a tempting target to try to expand into for some of those to set themselves apart.

[jstummbillig]

Assumedly, because they do know their place in the market and decided it's not great and that the outlook is worse.

Recently DHH of Rails fame gave a talk about Redis and how at their company it's being replaced in favor of long term storage on the now much faster and cheaper SSDs.

The takeaway was that the long caching it enables yields far better real world results in their products. He even had some charts.

[whalesalad]

> One main criticism of Redis had been its lack of support for SQL, the ubiquitous query language. Trollope said that was fixed now. A module RediSQL is available on GitHub.

I think redisql has been supplanted by zeesql - https://zeesql.com/

[Xeoncross]

I don't understand this. How can you want SQL for something that you are using because it's faster than SQL?

If your dataset fits in memory and you want relations then just use PostgreSQL/MySQL. Why would you pick the unique data structures and performance of redis sets/lists/pubsub/HLL and want SQL in front of it?

[iot_devs]

Ahahah

Author of zeesql/redisql here!

Unfortunately I was not able to figure out a clear business model for the module.

If any of you guys need it also the PRO version, just write me.

[ddorian43]

Pay $7/month for the license for each index/db after the first 3. I bet oracle ends up being cheaper.

[ohnoesjmr]

I've tried Redis-on-flash (enterprise version), and performance was very poor. Its just lsmt underneath and suffers from all the usual lsmt issues.

In the end we went with Aerospike which actually while backed by NVMe, outperforms redis backed by memory. Aerospike in memory is completely absurd speeds and throughput. Sadly the setup is a bit weird and query model is a bit wonky but we worked around it. A bit sad on how expensive the enterprise version is.

[karmakaze]

They mention both flash and disk, which finally gets explained as tiers 2 and 3. Also less clear though I assume this applies to their hosted service offering and not the core software.

> "We can take the lesser-used data that hasn't been touched in a while and shuttle it off to flash where it can sit for a while. When the user comes back eventually, it's very easy for us to seamlessly move it from flash back into memory. And that allows the company to save costs," he said.

> Redis is now planning to extend the concept to disk-based memory to offer support for a three-tiered architecture.

[SteveNuts]

I would like at least two more tiers: S3, and an LTO tape drive.

[iot_devs]

How bad is Redis with a swap file on SSD?

I would expect it to be a very reasonable middle ground that requires almost no work and it is already enough and ready for most use cases.

[reconditerose]

It's really bad. The problem is that the swapping of pages is constantly stalling the main Redis thread, which is not built for parallelism. Redis is built around a single highly efficient thread, so stalling it for 250us to pull a page from disk instantly reduces the throughput by an order of magnitude.

[otterley]

Anyone know how their plans differ from the Virtual Memory plans that antirez had about 13 years ago, implemented in Redis 2.6, and then (wisely) abandoned?

https://redis.io/docs/reference/internals/internals-vm/

[rjh29]

NVMe is a thing now.

[tehbeard]

How would the example of a scoreboard work with tiered memory?

Surely you need the sorted set in memory to derive rank/manipulate it etc?

Or would you have to shard it and add other elements on top in app code to focus on only a subsection of the data?

[pzmarzly]

> Surely you need the sorted set in memory to derive rank/manipulate it etc?

If you store it as an augmented balanced tree, then you only need to load the path from root to leaf to get or change rank. For example, if each node stored the number of leafs of its subtree, then you can avoid visiting most branches by simple logic: if you want to find element with rank=14, and left subtree only has 10 elements in total, ignore it and go straight to the right subtree, searching for element with rank=4 there.

[remram]

I think I would rather they added a write-ahead-log to make writes durable, than disk lookup to support bigger-than-RAM datasets (slowly).

[andrelaszlo]

It's not mentioned in the article but this is Redis Enterprise which is more like a superset of open source Redis.

[infomaniac]

memcached has recently gained the ability to spill to disk: https://github.com/memcached/memcached/wiki/Extstore

we recently implemented this to grow our caches to >50TB

[fumeux_fume]

Redisk... Sounds like a good April fools joke

[reconditerose]

It is part of the drive to make Redis "more like your classic database," he said. In the future, support for natural language queries and enhanced vector and feature store capabilities will be added.

Seems way off the mark for why people use Redis. Developers use it precisely because it's not like their classic databases.

[willsmith72]

The lines between data "stores" are blurring. I've seen it before where x team needs a fast key value store, so the obvious choice is redis. A year down the line, the product is evolving and the devs want to do some "classic" queries. Maybe it makes sense to keep it in redis based on their support for the queries you need, rather than migrating/replicating to a whole other system.

[dimgl]

I don’t think that’s necessarily always true. I was recently looking into using Redis as a regular database. Unfortunately data is still kept in memory even if it’s persisted on disk, so it was a no-go. You’d be surprised what people want!

[9dev]

What I actually need is a better way to bootstrap a cluster using Docker containers …

[wutwutwat]

https://redis.io/commands/restore/