Call me maybe: Aerospike

[Ankhers]

I currently work for a company that uses Aerospike quite heavily. In the past couple weeks, we have begun to notice data inconsistencies in our counters. We are seeing fluctuations in the data, despite having no decrement operations.

We have the enterprise edition of Aerospike, allowing us to be in constant contact with their support team and developers. A couple weeks later, and we still have no idea why this is happening. When dealing with monetary values, these fluctuations are very bad for us. Needless to say, we have begun migrating away from Aerospike.

[jboggan]

What is the rationale for storing monetary values in this sort of system? Not being snarky, just legitimately curious what scale of service could possibly necessitate that and what solutions didn't work beforehand.

[obituary_latte]

Can you add any detail to this anecdote? It's interesting and important, but detail might help steer others appropriately. What kind of inconsistency? What kind of fluctuation?

[cmpxchg16b]

This is interesting, and of course I have a couple of questions, but only two of them really matter: what client are you using, and are you using the cross-datacenter (XDR) replication functionality?

We* tested the increment functionality heavily (300K-1M aggregate ops/sec) before we turned it on in revenue service. We use it for a couple of different things, event counting is absolutely the major use case.

In a single-cluster world, it works phenomenally well. In a XDR world, things get a little tricky, and we had to change the way our application logic worked to compensate for it.

Any more information you can share about your use case?

*a big ad tech company that uses Aerospike heavily

[digitalzombie]

What are you migrating to ?

As is away from Aerospike to... ?

Thanks, very interesting anecdote/case.

[perlgeek]

When storing financial data, I'd certainly go with some kind of event sourcing: store deltas / financial transactions, not counters. The counters are just a sum over all deltas.

If performance is an issue, you can make the counters available in a second database that's only for reading, and updated from the original deltas.

[drvsrinivasan]

Today, the inconsistencies here were diagnosed by company and Aerospike to be caused by two clusters connected with XDR concurrently writing data to the same counter and shipping to each other and intentionally overwriting some data (bad design that somehow slipped through the cracks). So, this issue is unrelated to the Jepsen network partitioning tests that is the subject of the original article. The work that @Aphyr is doing is very valuable and much appreciated. (Aerospike Founder)

[smt88]

Yikes. What made you choose Aerospike in the first place?

[yellowapple]

When a product claims to have 100% uptime, I immediately cringe, knowing full well that they're probably full of bovine manure.

Good read.

[saryant]

Same with "exactly once delivery".

[grogers]

My analog to this is when someone claims that partitions are rare so you can ignore them, any product they make is very likely to lose data during partitions.

[peterwwillis]

https://www.youtube.com/watch?v=pjvQFtlNQ-M

[bketelsen]

Many kudos to Stripe for funding this. Truly a great gift to the community.

[mdellabitta]

I'm imagining the job interview.

"We want to sponsor your distributed database research, but not your Barbie animated GIF production."

[stephen_mcd]

The rotated A in the Aerospike logo reminds me of a system that's fallen over, and now you can't unsee it:

http://www.aerospike.com/

[drivers99]

Trying to read aerospike's front page to figure out what it is reminds me of this: http://shouldiuseacarousel.com/

[oska]

"A system"? Not sure what you mean by that.

To me, it was immediately apparent that they are making the whole word Aerospike look like a rocket, a motif they repeat through their home-page.

[AnDowNS]

My co-worker glanced at my screen and thought I was reading about erospike, which has a decidedly different feel, especially given the phallic imagery on the page.

[curun1r]

Really? When I look at it I only see an eye (viewed in profile) with the legs of the A being eye lashes.

[SchizoDuckie]

Oh wow, so this is not ACID, not "Eventual Consistency", but "Eventual Inconsistency"

[cschneid]

What is the 3 color chart that I've seen posted in several of these articles? I get that it breaks down the different CAP combinations with the things that it allows/implies. But is there a good breakdown of all the terms and what they mean?

[dwetterau]

Here is a paper [0] out of Berkeley that explains the chart you're asking about (the chart appears on page 8). For more information on each isolation level you might have to refer to the cited works or other sources.

[0] http://db.cs.berkeley.edu/papers/vldb14-hats.pdf

[gmagnusson]

I've used Aerospike at scale (approx 1MM tx per second) in private network, and smaller loads in cloud. I have always found it to be fast, reliable and extremely easy to operate (upgrade, modify cluster members, etc) w/o any downtime or interruption. It is a critical tool in my toolbox. I also have found their support and engineering team to be excellent.

I admire the work that Aphyr does - though at the end of the day, I need to build systems that work for the problem I'm trying to solve (and I have to choose from real things that are available).

Aerospike isn't the solution to every storage problem, and if you are choosing technology based on marketing material, you're probably going to be disappointed.

These technologies in general are trying to address really hard problems and design and architecture is the art of balancing tradeoffs. Nothing is going to be perfect. Yet.

[ploxiln]

"though at the end of the day" "nothing is perfect" ... Aerospike makes blatantly ridiculous promises in their high-level descriptions of their database. That makes our jobs harder (before Aphyr makes them easier) because we don't know what Aerospike is actually good at or exactly what kinds of data loss potential we need to architect our systems around.

Isn't it kind of annoying that some technical projects bolster their popularity/ecosystem with very fancy websites and impressive/competitive claims, but to really do your job right you have to throw all that away? The best you can do is try to get a sense from the reports of others who have tried something (and may or may not have been rigorous in their evaluation) so you can pick good candidates to even put through trials. (so again, thanks Aphyr)

[nemothekid]

While people tend to use Aphyr's posts as anti-nosql-cannon-fodder, I don't think he has ever advocated for anyone to straight up not use a database or used any of these results to show that no one should use NoSQL ever.

These seem like highly detailed Github Issues (infact the recent elasticsearch was a GH issue-turned-blog post), and these issue are brought to attention so that they could be fixed - not to slander the name of the company (and when they are, everyone benefits). IIRC, even after finding these bugs were published he continued to use elasticsearch.

Given how hard these problems are and how difficult they can be to reproduce, these writeups seem to be the most appropriate way to highlight these issues.

That said, if I was an aerospike user I'd be happier knowing this issue exists, someone has debugged it, and supported a detailed report about rather than being called in at 3am and discovering our data is funky.

[pkaye]

Fast, functional or reliable... pick two.

[me_again]

ITYM consistent, available or partition-tolerant...

[acqq]

I was wondering what is going on, with the titles of this form appearing on HN and now browsing the blog, it appears that the author is the fan of the "Call me maybe" titled posts (a lot of them there, and then also here!). From what I understand, this phrase, as used in the titles by him, seem to mean to him something like the "review of" something or the "comment on" something. For what it's worth.

[jack9]

The minimum useful licensing is in the tens of thousands of dollars. For the SLA they offer, it makes sense. Many well known Ad Serving companies utilize Aerospike (at a fraction of the cost of their previous solutions). It's very impressive result, per machine, from an operational standpoint.

[theVirginian]

"schemaless" nope

[tootie]

Is there any reason he's never tried to analyze a "classic" RDBMS like Oracle or SQL Server? I have to imagine they'd clobber a lot of this hipster technology.

[electrum]

He tested PostgreSQL: https://aphyr.com/posts/282-call-me-maybe-postgres

[samkone]

Lol He can't because Oracle licensing forbids to do this kind of work and make the results public.

[jhugg]

Well, those systems are largely not distributed, barring SQL Parallel Data Warehouse and, very arguably, RAC.

Jepson tests network partitions... so less useful.

[wglb]

Here is a bit of an index page: https://aphyr.com/tags/jepsen. It doesn't appear that he has. And yes, I would guess that they would indeed clobber those written there.

[elchief]

If you're big enough to require something like Aerospike, you're rich enough to build something like F1.

[vosper]

I dunno, there are ad-tech companies with less than 20 engineers who're processing millions of events per minute on their endpoints, and trying to do various things with that data. This from personal experience.

Of course, I would love if someone gave me the mandate to go out and build something like F1...

[rbetts]

You couldn't even hire a couple of experienced engineers let alone come close to affording to build F1. There are several zeros in the cost difference.

[nemothekid]

I can think of quite a few companies that are big enough to require something like Aerospike, but wouldn't be willing to outfit all their machines with a GPS receiver.

[peterwwillis]

Even if your company has 20 billion in cash, if your department isn't called "sales" or "marketing", good luck getting the budget for that.

And honestly, if you have money, it's a lot simpler and less risky to just hire 20 DBAs and programmers to build a database application that can handle that kind of operation. Low latency, network-partition-resistant, high-performance database applications are not a new thing.

[DAddYE]

Awesome read and still quite impressed by Aerospike.

[manigandham]

We use Aerospike heavily. It works just fine.

I'm constantly surprised by the general tone of comments on posts like these as if it's some crazy revelation that this software still obeys the fundamental laws of distributed systems.

There is no perfect database out there, all of them will fail with network partitions. Aerospike was designed to work in clusters that are very close together, often the same rack. It has much tighter timings and tolerances in exchange for providing much higher performance in certain situations and definitely has one of the best SSD focused storage systems I've come across.

If you don't have a high performance network interconnect between nodes, then there will be more issues with Aerospike since it relies on that more than some other system that use Paxos for all writes (like aphyr mentions). We run several TB's of data accessed at 100k+ TPS including very fine grained counters and everything works. And yes, we run on the cloud in AWS and SoftLayer and have yet to have major problems with the proper network setup.

Btw, there is a comment below from the current CTO of AppNexus, one of the companies that pioneered real-time bidding for digital ads and runs several million auctions per second on one of the biggest ad exchanges available. They were the first customer for Aerospike and from everything I've learned from their team, it works really well for them, and they definitely are not happy to just "lose" data however insignificant it might seem. Volume changes everything and even a fraction of a percent will add up. We trust Aerospike because it's been hardened by lots of much much larger companies with very high production usage, the key is being aware of all the technical requirements and the environment you're deploying in.

I think the real major issue here that people seem upset with are the general claims and marketing information. I can't speak to all that and there are definitely some things like 100% uptime which do seem overly confident, but this is true of every single technology vendor out there unfortunately. I'm not saying Aerospike is any better or worse as a company but marketing material only goes so far and it would surprise me if further research wasn't done for any mission critical system.

[teraflop]

> There is no perfect database out there, all of them will fail with network partitions.

Some of them will fail in a way that keeps your data safe, others will fail in a way that preserves uptime but gives you temporarily inconsistent data. Aerospike apparently does neither. Why is it unreasonable to expect them not to falsely claim otherwise?

The "crazy revelation" for me was not that Aerospike's software is, like everything else, subject to the CAP theorem. It's that they apparently think it's awesome to claim that it isn't, and charge tens of thousands of dollars for their product on that justification.

[obstinate]

> We trust Aerospike because it's been hardened by lots of much much larger companies with very high production usage

Why would you trust something after its untrustworthiness is demonstrated before your eyes? Just because some other companies use it and haven't yet publicized dissatisfaction? That is not how you make sound engineering decisions.

[pjc50]

Genuine question: how do you know you've never lost a single write?

[woozy]

I adore its SSD read/write optimization. The rest were understood as marketing materials.