Dynomite – Making Non-Distributed Databases Distributed

[void_star]

It's a little unfortunate that consistency is only mentioned as a footnote. If this is being used internally at Netflix, how are they dealing with consistency issues in the face of failed or conflicting writes?

[antirez]

This currently looks like the Dynamo-alike sharding without the read-repair part making values eventually convergent in all the replicas. This latter feature is much needed, but they are already planning it apparently.

[halayli]

If it's based on Dynamo, then it's using vector clocks.

[nteon]

Netflix is a Java-heavy shop (the client talked up in the blog post is the Java client). I was pleasantly surprised to see that dynomite is implemented in C with an eye towards both BSD + Linux. I'm curious if this was because a GC'ed language like Java or Go was tried and proven not up to the task, or what.

[viksit]

My guess would be that the ecosystem in their (seed) target set of databases and protocols is written in C. In one case, they're using Twitters twemproxy to provide connection counts for redis/memcached.

[mumrah]

My guess would be that they went with C to make it easier to hook into existing data systems like MySQL or Redis.

[padho]

My guess is that this has a performance reason

[personZ]

Netflix always makes for an interesting case, but what always strikes me is how incredibly "trivial" their application really is: It's a simple application at a very large scale. This doesn't dismiss their accomplishments, of course, but often it does seem like Netflix spends an enormous engineering effort on fairly low value aspects of the service.

[themartorana]

Well, they make it seem trivial, while dealing with every watch and every pause, every rating updating complex recommendation engine algorithms. They deal with licensing and region awareness and proxying by ISP and running their own CDNs and...

And that's the magic.

To boot, they do it at enormous scale on a multitude of platforms all while their own destructive code (Chaos Monkey) runs around destroying servers.

It's true, some things like "stream me this" are maybe not rocket science, but at Netflix scale, everything becomes a bit more difficult.

[jrallison]

You're discounting the "very large scale" portion of your statement.

What may seem fairly low value from the outside or consumer perspective is likely of huge value internally to ensure you can always watch Netflix without even thinking about everything that has to happen to deliver as much video as they do to their customer's browsers.

[morgante]

Comments like this really annoy me.

What exactly is trivial about building a complicated multisource streaming video service with sophisticated recommendations (plus all the other "little" stuff like payments)?

Google is "trivial." It's just a box for doing text search on a graph database. Apple is "trivial." They just make pretty skins for commodity hardware & software. Etc.

When you consider 99% of everything in your field trivial, maybe you should consider refining your definition of trivial.

[chaz]

Netflix currently has 94 open engineering open reqs, plus another 22 data/analytics open reqs. Perusing through the list should give you a sense of how complicated their products are: http://jobs.netflix.com/jobs.php

[jeangenie]

An idea's utility is often orthogonal to its complexity.

[brown9-2]

this seems like a pretty common fallacy for people to think about an organization from the outside

[jedberg]

When I saw this internally a few months ago, I said that I haven't been more excited about a new project in a long time. I personally think this will solve a lot of scalability problems people have while letting them keep their favorite database.

[stefanwild]

It looks very promising. I like the fact that it doesn't only work for AWS environments.

[lobster_johnson]

Funnily enough Dynomite was also the name of an Erlang implementation (https://github.com/moonpolysoft/dynomite) of Dynamo that seemed to get off to a very promising start before it was abandoned a few years ago.

[NKCSS]

Cool, it's something that Microsoft SQL Server (Enterprise edition) offers out of the box (Peer-2-Peer replication). The only problem with P2P replication is that, at large scale, requires massive amounts of bandwidth to keep everything synced, and it gets worse the more nodes you add.

[diptanu]

Depends how things are replicated. In Dynamo style replication everything isn't replicated everywhere. Based on consistent hashing, and how vnodes etc are configured, a key lands itself in more than one nodes so that things are generally available. So scale-out in a dynamo-like system happens more gracefully.

[ppearcy]

Any ideas if this supports Postgres?

Also, I noticed comments about an architecture document here: https://github.com/Netflix/dynomite

Anyone know where the doc they are referencing is located?

[jedberg]

It doesn't support Postgres yet but that is on the (longer term) roadmap.

However, since it is open source, contributions are welcome! :)

[diltonm]

I don't know if it was intentional but the name reminds me of the character JJ in the TV series Good Times. Awesome show that was. It sounds solid overall but one thing caught my eye, "A client can connect to any node to read the data. Similar to writes, a node serves the read request if it owns the data, otherwise it forwards the read request to the data owning node in the same rack.". I wonder if that could lead to an owning node arbitration storm when more than one node in a rack owns the data?

[fizx]

For projects with similar goals, see also Twitter's Gizzard (abandoned) and LinkedIn's Voldemort.

[anonfunction]

Add Mashape's Astronode to the list: https://github.com/Mashape/Astronode-Broadcaster

[wowmsi]

Also Riak. However, one interesting aspect of Dynomite seems to be offer default support for clustering across multiple data centers with pluggable stores.

[RealCasually]

Perhaps I missed it, but is there any discussion on how their sharding mechanism works? How do they define a shard and where is ownership of various shards managed so a "non-owning node can redirect to an owning node"?

[marktangotango]

I questioned tbat as well, could be a property of the underlying store (memcache hash). How would mysql sharding work without being application managed?

[jlafon]

It's interesting that Netflix decided to write this rather than using Amazon's own DynamoDB. I wonder specifically if DynamoDB was too expensive (as I have found), or was there some other reason?

[jetblackio]

This is completely different than DynamoDB. Essentially this is a framework for sharding or horizontally scaling multiple datastores (MySQL, Memcache, Redis, etc).

[grogers]

Keep in mind that they define

DC := AWS region

Rack := AWS AZ

That should tell you that they need multi-region availability, something that DynamoDB will likely never provide.

[fizx]

This is the sort of project you do to get cross-region replication on your existing data stores, many of which look more like PostgreSQL than KV stores.

[halayli]

nitpick, the code check for NULLs at a low level but misses checking NULLs at higher levels like here:

https://github.com/Netflix/dynomite/blob/master/src/dyn_dnod...

[evantahler]

Are there any deployment notes/examples?

[nrbafna]

Setup and Installation Wiki - https://github.com/Netflix/dynomite/wiki/Getting+Started#set...

[raxen]

not really, the best you are going to find so far: https://github.com/Netflix/dynomite