[krka]

Yes, this post was handwaving, and I tried to make that clear ("preliminary", "extremely biased").

On monday I added some slightly more proper benchmark code, you can find it on https://github.com/spotify/sparkey/blob/master/src/bench.c

I didn't add the level-db code to this benchmark however, since I 1) didn't want to manage that dependency 2) didn't know how to write optimized code for usage of it.

I'm using very small records, a couple of bytes of key and value. The insert order is strictly increasing (key_0, key_1, ...), though that doesn't really matter for sparkey since it uses a hash for lookup instead of ordered lists or trees.

As for the symas mdb microbench, I only looked at it briefly but it seems like it's not actually reading the value it's fetching, only doing the lookup of where it actually is, is that correct?

"MDB's zero-memcpy reads mean its read rate is essentially independent of the size of the data items being fetched; it is only affected by the total number of keys in the database."

Doing a lookup and not using the values seems like a very unrealistic usecase.

Here's the part of the benchmark I'm referring to: for (int i = 0; i < reads_; i++) { const int k = rand_.Next() % reads_; key.mv_size = snprintf(ckey, sizeof(ckey), "%016d", k); mdb_cursor_get(cursor, &key, &data, MDB_SET); FinishedSingleOp(); }

[hyc_symas]

It's a valid measurement of how long the DB takes to access a record. You can assume that the time required for an app to use the values will be the same across all DBs therefore it's omitted. All of the microbench code behaves the same on this point.

http://symas.com/mdb/memcache/ and http://symas.com/mdb/hyperdex/ give results where the records are transmitted across the network.

[krka]

Well, some DB's "load" the value in some way before giving it to the user, so that time is implicitly measured for those types of DB's but not for others, so I don't think it's a particularly fair comparison. I think Tokyo Cabinet gives you a pointer to newly allocated memory, at least for compressed data (but I am not completely sure about this). Like LMDB, Sparkey also does no processing of the value for uncompressed data, but for compressed data some decompression needs to take place in the iterator buffer (I guess that's equivalent to your cursor object). Even worse, if this is done lazily upon value retrieval, the cost is completely hidden in the benchmark.

In any case, I think the easiest way to get a fair benchmark is to at least iterate over the value, possibly also compare it. If that time turns out to be significant (perhaps even dominant) compared to the actual lookup time, then further optimization of the actual storage layer is pretty meaningless.