Optimizing your CouchDB Calls by 99% (no strings attached)

[wccrawford]

I'll save everyone the time:

Don't use CouchRest::Database! because it tries to create the DB each time. (Use CouchRest::Database instead)

Don't let multi_json pick the json library! Pick jsonx yourself before couchrest runs.

Don't use HTTPS for internal connections.

For the last 3%... He hacked in a binary format into couchrest and couchdb in 10 lines of code.

[pavel_lishin]

Thank you. I'm too ADD for videos, and besides, don't know of an easy way to import them into iTunes as a podcast-style video (one that won't restart from the beginning if I stop it halfway through.)

[rkalla]

I like the idea that Tim is working on a binary-based protocol interaction with Couch. If the binary protocol integration is generic enough I'd like to discuss with them the possibility of supporting the Universal Binary JSON format (http://ubjson.org)

While the HTTP-only protocol work up until now has been one the nice "Ease of use" standout features for Couch, it also leads to a lot of overhead when you know you absolutely don't benefit from it.

All the work on CouchApps (only CouchDB + JavaScript apps) makes a lot of sense to have the rest-only interaction, but if you are trying to scale out a high read/write couch install and are already using tricks like TCP no_delay and server-side batching along with batch-doc operations on the client there is only so much you can do before you slam up against what looks to be a 6-10K docs/sec change rate due in no small part to Couch (even with its new JSON parser) processing data structures into and out of textual JSON.

[timanglade]

To be clear, I'm only working on integrating a binary data format, which is different from a binary protocol.

HTTP is baked pretty deep into CouchDB, which makes it hard to remove (but who knows, Couchbase seems to be having some success ripping it out). BigCouch (Cloudant’s distributed version of CouchDB) has the HTTP layer neatly separated through the Fabric library (https://github.com/cloudant/fabric), so it'd probably be easier to add a binary protocol there.

[mlmilleratmit]

Good point. A binary protocol is certainly the ultimate play. Tim has some promising initial estimates, but the gains depend on the document properties (mostly size). In the meantime an alternative solution is to spread the server-side load via BigCouch or some other clustering solution. However, I'd wager that even with slow JSON libs most workflows are still IO limited.

[dochtman]

Are there no slides or transcript, or blog posts? It would be nice to have something that can be skimmed through.

I don't use Ruby, so I'm not sure how much of the talk would apply to me, but we use CouchDB extensively at work.

[rkalla]

Are you guys using any of the replication? If so, have you seen any of the replication failures that are tracked in JIRA right now?

There are 2 or 3 nebulous "replication breaks" bugs that are filed from older versions, but one that says it is still happening on 1.1.1 with no followups from the team.

Filipe I believe rewrote the replication code from scratch for 1.2 so again, it is hard to know if the replication hickups are non-issues or not so I'm curious if you have real world data points here.

[andrewvc]

I have to wonder why so many presentations have 'X sucks, I say use Y which rocks' as an attitude. Negativity doesn't really sell me on much.

I realize he has just grievances against multi-json and couch-rest, but really that kind of attitude drags the whole community down.

OSS should be about collaboration, not a dick swinging contest.

[timanglade]

I have to wonder why so many comments have ‘X sucks, please say Y which rocks’, as an attitude. Negativity doesn't really sell me on much.

Seriously though. The poor design (& egregious misuse) of those libraries became a major concern for us at Cloudant. I wanted people to become aware of what they were doing wrong & why. I thought it would be best done with a bang. Fixing the situation beyond the patches I submitted would have required that I go on a campaign to have those libraries removed from the face of the earth (which seems more extreme than very strongly recommending people stay away from them).

[cloudhead]

Mostly a waste of time.. there's nothing about optimization, just common sense, like "don't parse json with ruby".

[baudehlo]

So in summary the performance issue was entirely with JSON encoding/decoding.

[Will_Price]

I know this is going to sound pedantic, however, optimizing by 99% would make them 1% less efficient? Right? Or am I muddled?

[cloudhead]

optimization is inherently positive, so it could either mean +99% performance, ie 199% or -99% time.