So I assume this means cockroachDB currently (probably) meets its promised consistency levels. How does it compare to the usual default settings of PostgreSQL for example? - I think you get SERIALIZABLE, so the behaviour should be very reasonable.
I understand the txn/s numbers are in a pathological scenario, but will these transactions block the whole db from moving faster or have unrelated transactions still normal throughput?
I'm just glad we get linearizable single keys, I had to switch to HBase because it was the only choice at the time. Cassandra did not even give me read-your-own-writes. HBase also had issues while testing on the same host, delete+write lead to deletes being reordered after the writes because they had the same timestamp.
I'm a bit disappointed from HN in this post, everyone only talks about the name. But 90% probably did not even read the article.
Aaanyway, heres my 2cents: https://cr.yp.to/cdb.html is CDB for me :P - so better choose another acronym!
> How does it compare to the usual default settings of PostgreSQL for example? - I think you get SERIALIZABLE, so the behaviour should be very reasonable.
Not clear if you're referring to PostgreSQL or CockroachDB, but for the record, the PostgreSQL default transaction isolation level is READ COMMITTED, not SERIALIZABLE.
(Cockroach Labs CTO) CockroachDB provides two transaction isolation levels: SERIALIZABLE, which is our default and the highest of the four standard SQL isolation levels, and SNAPSHOT, which is a slightly weaker mode similar to (but not quite the same as) REPEATABLE READ. Unlike most databases which default to lower isolation modes like REPEATABLE READ or READ COMMITTED, we default to our highest setting because we don't think you should have to think about which anomalies are allowed by different modes, and we don't want to trade consistency for performance unless the application opts in.
All transaction interactions are localized to particular keys, so other transactions can proceed normally while there is contention in an unrelated part of the database.
(as for the acronym, we prefer CRDB instead of CDB)
I don't believe CockroachDB provides SERIALIZABLE consistency for the entire keyspace, rather for subsets of the keyspace that reside in the same Raft cluster.
In practice this means no cross key serializability without including a serialization token in transactions.
I may be mistaken however but that is my understanding from reading CockroachDB docs and Aphyrs post.
"For instance, on a cluster of five m3.large nodes, an even mixture of processes performing single-row inserts and selects over a few hundred rows pushed ~40 inserts and ~20 reads per second (steady-state)."
Jepsen is essentially a worst-case stress test for a consistent database: all the transactions conflict with each other. CockroachDB's optimistic concurrency control performs worse with this kind of workload than the pessimistic (lock-based) concurrency control seen in most non-distributed databases. But this high-contention scenario is not the norm for most databases. Most transactions don't conflict with each other and can proceed without waiting. Without contention, CockroachDB can serve thousands of reads or writes per second per node (and depending on access patterns, can scale linearly with the number of nodes).
And of course, we're continually working on performance (it's our main focus as we work towards 1.0), and things will get better on both the high- and low-contention scenarios. Just today we're about to land a major improvement for many high-contention tests, although it didn't speed up the jepsen tests as much as we had hoped.
a.) this is a beta product where the team has been focusing on completeness and correctness before performance
b.) I was testing Cockroach from August through October--Cockroach Labs has put a good deal of work into performance in the last three months, but that work hasn't been merged into mainline yet. I expect things should be a good deal faster by 1.0.
c.) Jepsen does some intentionally pathological things to databases. For instance, we choose extremely small Cockroach shard sizes to force constant rebalancing of the keyspace under inserts. Don't treat this as representative of a production workload.
Even if they manage to multiply this by 100 on the final release, it's still way weaker than a regular sql db. I hope they have another selling point than performance.
Like Spanner, Cockroach’s correctness depends on the strength of its clocks... Unlike Spanner, CockroachDB users are likely deploying on commodity hardware or the cloud, without GPS and atomic clocks for reference.
I'm curious, do any of the cloud providers offer instances with high-precision clocks? Maybe this is something one of the smaller cloud providers might want to offer to differentiate themselves - instances with GSP clocks.
The spanner announcement mentioned offering direct access to the TrueTime api sometime in the not too distant future.
Edit: Quote from Brewer's post: "Next steps
For a significantly deeper dive into the details, see the white paper also released today. It covers Spanner, consistency and availability in depth (including new data). It also looks at the role played by Google’s TrueTime system, which provides a globally synchronized clock. We intend to release TrueTime for direct use by Cloud customers in the future."
> I'm curious, do any of the cloud providers offer instances with high-precision clocks?
Essentially that's what Google's doing by selling Spanner, but you don't get access to the clocks directly. I'd guess that as they get more experience selling Spanner, and they bring the price of the hardware down, they'll rent those directly too.
>I'm curious, do any of the cloud providers offer instances with high-precision clocks?
That's a great question.
Could be a little tricky with AWS for example. It's typical to run a database on a VPC (virtual private cloud), since that layer of the stack doesn't need to be exposed to the Internet. Unfortunately, that means the servers in the VPC can't get to the ⁎.amazon.pool.ntp.org servers.
Skimming around, I don't see any of the cloud providers, VPS providers, or dedicated hosting companies offering anything special, like a local stratum 1 time source for each region/location.
Wow. Probably fundraising and getting out front of googles new consumer spanner service. I just started experimenting w/ cockroachdb and it sounds/seems great. They def are campaigning hard, landing a Jepsen test and a lot of posts/articles last couple.
I am pulling for them over G on principle. I infer they know people at Stripe where the Rethink team & Aphyr are so hopefully they can learn from them and build an awesome product. G has lost a lot of battles on UX, Docs, attitude and just general product. They have won a lot of battles on sheer size & resources. Be nice to have this stick around.
CDB is a really cool db for the time I've been playing with it (just a few months I think).
That said, I'd really appreciate if they'd launch a hosted DBaaS. By that I mean, more in line with dynamodb or documentdb instead of say MySQL on AWS for example.
I know it's on their timeline, but dbs generally get selected during the start of the projects and unless absolutely necessary, companies don't want to change dbs.
CockroachDB employee here. DBaaS is definitely on our roadmap. There are a lot of moving pieces to building out DBaaS, so our focus first is on honing our core product. In the meantime, we are working on making it as easy as possible to deploy CockroachDB across different cloud vendors and with orchestration tools like Kubernetes.
> Should the clock offset between two nodes grow too large, transactions will no longer be consistent and all bets are off.
A concern I have is that there could be datacorruption in this context, no? It sounds like on a per cluster basis, i.e. Same Raft cluster, there is a strict serializability guarantee. But without linearalizability system wide, you could end up with data corruption on multi-keyspace writes.
You're correct--large clock anomalies (or VM pauses, message delays, etc) can cause serializability anomalies. I don't exactly know what will happen, because our tests explicitly focused on staying within clock skew bounds.
To speculate: if clocks do go out of bounds, I don't... think... you can violate single-key serializability, because that goes through a single Raft cluster, but you can definitely see multi-key consistency anomalies. I think you could also get stale single-key reads, because the read path IIRC relies on a time-based lease. Might be other interesting issues if Cockroach is shuffling data from one Raft ensemble to another--I think the range info itself might have a time-based lease that could go stale as well. Perhaps a Cockroach Labs engineer could comment here?
Note that the time window for these anomalies is limited to a few seconds-nodes will kill themselves when they detect clock skew.
Of course anything good may eventually rise to success on its merits, but in case I'm not the only one with a subconscious aversion maybe the info below will help put some people one less click away. How about SurvivalDB? :)
CDB is a distributed SQL database built on a transactional and strongly-consistent key-value store. Scales horizontally; survives disk, machine, rack, and datacenter failures with low latency disruption and no intervention; strongly-consistent ACID transactions; SQL API for querying data. Inspired by Google’s Spanner and F1. Completely open source.
Absolutely the name is holding them back. In my case it prevents me from championing the product within my company. Going to the executives and saying "we're going to put our most precious data into Cockroach" simply won't fly. Any discussion of features will be for naught; I know the CEO will make a "no" decision as soon as he hears the name.
To any Cockroach Labs folks who might read this: please give me (and all of us) a name that I can evangelize without my audience cringing.
I agree completely. I love reading about databases, but apparently have been skipping reading about CockroachDB for that same reason subconsciously. I took a deep dive the other day and was amazed that I've seen the name dozens of times and have never bothered to ask what was special about it. It really is a fascinating database.
It's a terrible name, it also has the property that stupidly configured web filters could filter it due to string matching the first part of the word.
It also makes your employees look stupid when they tell their landlords, bank accounts, dates that they work at Cockroach Labs. Good luck explaining that you aren't an exterminator.
I submit "Bunker" as a more approachable name. Not only does it have the connotation of "survives nuclear bombardment", but also suggests "keep your stuff safe here" and "something you could entrust with your life".
Agree 100%, ditto for suggested alternate name. I'd been doing the same for Rust. It's not a rational reason, sure, but I have to sell adoption to a broader userbase that cares about more that the technical merits.
I don't want to have to make the case for why someone's system needs rust and cockroaches.
I'm wondering if all the people who can't get over the name CockroachDB are also people who insist that professionals should dress the part. Can't take someone seriously if they don't have their shirt tucked in?
Yeah, I've definitely heard people tell me "It seems cool, but not cool enough for me to get over the subconscious revulsion to the name" when asking about CDB.
I appreciate the name, but I think I'd have an easier time convincing people to use a product of a different name....
It might be intentional as reverse psychology. They don't want employees or users who wouldn't use a project because it's named after an unpopular insect. Those users/employees could be fickle, unreasonable and low quality at other aspects of business/development and be a net loss for the company/project in the long term.
Completely agree. I specialize in distributed databases and I have zero interest in even learning about CockroachDB. Could be the greatest technology ever and I will never know. Is that irrational? Sure. But I'm a busy man with plenty of tech to learn about and limited time. I detest roaches and do not want to think about them at all when I am working. Might as well have named it with an ethnic slur.
to me it's not a deeply-seeded reason, it's just when I hear cutesy-sounding names I just don't take them seriously. They sound like the latest, hipster thing that's going to be in-bloom for a (figurative) week or two until the next cutesy thing steals everyone's attention. Even the language Pony has a mildly off-putting name (to me) despite hearing people rave about it. Couch DB ? What the hell does that mean ?
I admit it's a bias of mine and I could be losing out on learning amazing tools. But even non-profit, standardized languages and tools still have a brand name and image.
their name is the stupidest startup blunder I've ever seen. Enterprise won't go near it, seriously. I cannot tell a client I'm using a "cockroach" database, no fucking way.
If we could go back and rename Windows Hitler would anyone be using it???
I wish everyone would stop bikeshedding the damn name.
From my perspective I'm glad something that's trying to do this is coming into existence. a couple years ago I was given the arduous task of ensuring that we never lost data. One of the requirements was that: At any time, any server can (and will) fail.. if you acknowledge that you have data then you MUST never lose it.
You cannot imagine how many database solutions that ruled out. Including basically _all_ noSQL data stores.
We settled on postgresql, because, despite going into toast tables and having to implement sharding on top- it was not only the most sane it could be wrangled into basically doing the right thing in regards to fsyncing and ensuring data wasn't in VFS.
So, Kudos to these guys, who, in a world of people who don't give a shit about consistency (except eventual consistency) are actually trying to further it.
I'm disappointed with the commentary here, I for one would think the discussions around building a distributed system with the consistency guarantees claimed here would be far more interesting than the name of said system.
CockroachDB might be the worst product name I've come across. I get it; cockroaches are extremely hardy, but they're also much more strongly associated with uncleanliness and disease.
EDIT: Also, cockroaches are BUGS! Bugs cause problems in computer systems.
[+] [-] ysleepy|9 years ago|reply
I understand the txn/s numbers are in a pathological scenario, but will these transactions block the whole db from moving faster or have unrelated transactions still normal throughput?
I'm just glad we get linearizable single keys, I had to switch to HBase because it was the only choice at the time. Cassandra did not even give me read-your-own-writes. HBase also had issues while testing on the same host, delete+write lead to deletes being reordered after the writes because they had the same timestamp.
I'm a bit disappointed from HN in this post, everyone only talks about the name. But 90% probably did not even read the article. Aaanyway, heres my 2cents: https://cr.yp.to/cdb.html is CDB for me :P - so better choose another acronym!
[+] [-] ak4g|9 years ago|reply
Not clear if you're referring to PostgreSQL or CockroachDB, but for the record, the PostgreSQL default transaction isolation level is READ COMMITTED, not SERIALIZABLE.
https://www.postgresql.org/docs/current/static/sql-set-trans...
[+] [-] bdarnell|9 years ago|reply
All transaction interactions are localized to particular keys, so other transactions can proceed normally while there is contention in an unrelated part of the database.
(as for the acronym, we prefer CRDB instead of CDB)
[+] [-] jpgvm|9 years ago|reply
In practice this means no cross key serializability without including a serialization token in transactions.
I may be mistaken however but that is my understanding from reading CockroachDB docs and Aphyrs post.
[+] [-] BrentOzar|9 years ago|reply
Wow, that's...not fast.
[+] [-] bdarnell|9 years ago|reply
Jepsen is essentially a worst-case stress test for a consistent database: all the transactions conflict with each other. CockroachDB's optimistic concurrency control performs worse with this kind of workload than the pessimistic (lock-based) concurrency control seen in most non-distributed databases. But this high-contention scenario is not the norm for most databases. Most transactions don't conflict with each other and can proceed without waiting. Without contention, CockroachDB can serve thousands of reads or writes per second per node (and depending on access patterns, can scale linearly with the number of nodes).
And of course, we're continually working on performance (it's our main focus as we work towards 1.0), and things will get better on both the high- and low-contention scenarios. Just today we're about to land a major improvement for many high-contention tests, although it didn't speed up the jepsen tests as much as we had hoped.
[+] [-] aphyr|9 years ago|reply
a.) this is a beta product where the team has been focusing on completeness and correctness before performance
b.) I was testing Cockroach from August through October--Cockroach Labs has put a good deal of work into performance in the last three months, but that work hasn't been merged into mainline yet. I expect things should be a good deal faster by 1.0.
c.) Jepsen does some intentionally pathological things to databases. For instance, we choose extremely small Cockroach shard sizes to force constant rebalancing of the keyspace under inserts. Don't treat this as representative of a production workload.
[+] [-] tveita|9 years ago|reply
[+] [-] sametmax|9 years ago|reply
[+] [-] exhilaration|9 years ago|reply
I'm curious, do any of the cloud providers offer instances with high-precision clocks? Maybe this is something one of the smaller cloud providers might want to offer to differentiate themselves - instances with GSP clocks.
[+] [-] mikecb|9 years ago|reply
Edit: Quote from Brewer's post: "Next steps For a significantly deeper dive into the details, see the white paper also released today. It covers Spanner, consistency and availability in depth (including new data). It also looks at the role played by Google’s TrueTime system, which provides a globally synchronized clock. We intend to release TrueTime for direct use by Cloud customers in the future."
[+] [-] BrentOzar|9 years ago|reply
Essentially that's what Google's doing by selling Spanner, but you don't get access to the clocks directly. I'd guess that as they get more experience selling Spanner, and they bring the price of the hardware down, they'll rent those directly too.
[+] [-] tyingq|9 years ago|reply
That's a great question.
Could be a little tricky with AWS for example. It's typical to run a database on a VPC (virtual private cloud), since that layer of the stack doesn't need to be exposed to the Internet. Unfortunately, that means the servers in the VPC can't get to the ⁎.amazon.pool.ntp.org servers.
Skimming around, I don't see any of the cloud providers, VPS providers, or dedicated hosting companies offering anything special, like a local stratum 1 time source for each region/location.
[+] [-] Already__Taken|9 years ago|reply
[+] [-] vonklaus|9 years ago|reply
I am pulling for them over G on principle. I infer they know people at Stripe where the Rethink team & Aphyr are so hopefully they can learn from them and build an awesome product. G has lost a lot of battles on UX, Docs, attitude and just general product. They have won a lot of battles on sheer size & resources. Be nice to have this stick around.
[+] [-] aphyr|9 years ago|reply
[+] [-] AsyncAwait|9 years ago|reply
1 - https://github.com/pingcap/tidb
[+] [-] ohstopitu|9 years ago|reply
That said, I'd really appreciate if they'd launch a hosted DBaaS. By that I mean, more in line with dynamodb or documentdb instead of say MySQL on AWS for example.
I know it's on their timeline, but dbs generally get selected during the start of the projects and unless absolutely necessary, companies don't want to change dbs.
[+] [-] dianasaur323|9 years ago|reply
[+] [-] bluejekyll|9 years ago|reply
A concern I have is that there could be datacorruption in this context, no? It sounds like on a per cluster basis, i.e. Same Raft cluster, there is a strict serializability guarantee. But without linearalizability system wide, you could end up with data corruption on multi-keyspace writes.
[+] [-] aphyr|9 years ago|reply
To speculate: if clocks do go out of bounds, I don't... think... you can violate single-key serializability, because that goes through a single Raft cluster, but you can definitely see multi-key consistency anomalies. I think you could also get stale single-key reads, because the read path IIRC relies on a time-based lease. Might be other interesting issues if Cockroach is shuffling data from one Raft ensemble to another--I think the range info itself might have a time-based lease that could go stale as well. Perhaps a Cockroach Labs engineer could comment here?
Note that the time window for these anomalies is limited to a few seconds-nodes will kill themselves when they detect clock skew.
[+] [-] mixedCase|9 years ago|reply
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] WhitneyLand|9 years ago|reply
I only recently read some cool things about CDB and wondered if I could have been subconsciously skipping articles about it for deep seeded reasons like http://bbc.com/future/story/20140918-the-reality-about-roach...
Of course anything good may eventually rise to success on its merits, but in case I'm not the only one with a subconscious aversion maybe the info below will help put some people one less click away. How about SurvivalDB? :)
CDB is a distributed SQL database built on a transactional and strongly-consistent key-value store. Scales horizontally; survives disk, machine, rack, and datacenter failures with low latency disruption and no intervention; strongly-consistent ACID transactions; SQL API for querying data. Inspired by Google’s Spanner and F1. Completely open source.
https://cockroachlabs.com/docs/frequently-asked-questions.ht...
[+] [-] jdcarter|9 years ago|reply
To any Cockroach Labs folks who might read this: please give me (and all of us) a name that I can evangelize without my audience cringing.
[+] [-] finder83|9 years ago|reply
[+] [-] influx|9 years ago|reply
It also makes your employees look stupid when they tell their landlords, bank accounts, dates that they work at Cockroach Labs. Good luck explaining that you aren't an exterminator.
[+] [-] Birch-san|9 years ago|reply
[+] [-] SilasX|9 years ago|reply
I don't want to have to make the case for why someone's system needs rust and cockroaches.
[+] [-] justicezyx|9 years ago|reply
Otherwise, its a realistic issue has to be addressed.
[+] [-] OpenDrapery|9 years ago|reply
Style over substance.
[+] [-] geofft|9 years ago|reply
I appreciate the name, but I think I'd have an easier time convincing people to use a product of a different name....
[+] [-] ArlenBales|9 years ago|reply
[+] [-] wstrange|9 years ago|reply
[+] [-] Fiahil|9 years ago|reply
A really bad name, would be, for example, to try calling a news agency "BuzzFeed News".
[+] [-] renegadesensei|9 years ago|reply
[+] [-] crudbug|9 years ago|reply
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] _mg|9 years ago|reply
[+] [-] Perignon|9 years ago|reply
[+] [-] digler999|9 years ago|reply
I admit it's a bias of mine and I could be losing out on learning amazing tools. But even non-profit, standardized languages and tools still have a brand name and image.
[+] [-] throwawasiudy|9 years ago|reply
If we could go back and rename Windows Hitler would anyone be using it???
[+] [-] elcct|9 years ago|reply
[deleted]
[+] [-] dijit|9 years ago|reply
From my perspective I'm glad something that's trying to do this is coming into existence. a couple years ago I was given the arduous task of ensuring that we never lost data. One of the requirements was that: At any time, any server can (and will) fail.. if you acknowledge that you have data then you MUST never lose it.
You cannot imagine how many database solutions that ruled out. Including basically _all_ noSQL data stores.
We settled on postgresql, because, despite going into toast tables and having to implement sharding on top- it was not only the most sane it could be wrangled into basically doing the right thing in regards to fsyncing and ensuring data wasn't in VFS.
So, Kudos to these guys, who, in a world of people who don't give a shit about consistency (except eventual consistency) are actually trying to further it.
[+] [-] irfansharif|9 years ago|reply
[+] [-] tshannon|9 years ago|reply
[+] [-] will_pseudonym|9 years ago|reply
EDIT: Also, cockroaches are BUGS! Bugs cause problems in computer systems.
[+] [-] sroussey|9 years ago|reply
[+] [-] throwawasiudy|9 years ago|reply
How about Duradero ... Spanish for durable. The name doesn't seem to be taken by any other database type software