top | item 9782520

Kerf: a columnar tick database for Linux, OS X, BSD, iOS, Android

62 points| luu | 10 years ago |github.com

30 comments

order
[+] CyberShadow|10 years ago|reply
Do I understand correctly that this is proprietary (closed-source) software, even though it's hosted on GitHub? This repository seems to contain two binary blobs without source code.
[+] paulasmuth|10 years ago|reply
I don't get all the negativity RE: kerf not being open source in this thread. It's obvious the implementation is not free software (check the README in the actual submission) because the author wants to sell commercial licenses. Nothing inherently bad about that -- some of us need to make a living.

The query language and concepts which are explained in the documentation are an exciting/novel (at least to me) contribution and "out in the open" for everybody to see/discuss/adapt/improve upon. Let's not dismiss that because we can't read the source.

[+] Daemon404|10 years ago|reply
Nothing bad ever happens to proprietary database software... nope.
[+] paulasmuth|10 years ago|reply
Off topic; It looks like you linked the linux binary completely statically. I am curious why you decided to do this and how you achieved it. Did you use something other than glibc to compile these binaries?

Also did you roll the SQL engine from scratch or does it built on some existing parser/runtime? Would love to learn more.

[+] kcl|10 years ago|reply
Linux linking: I wasn't doing anything special with the linking. The Linux binary was compiled on a VirtualBox instance of Mint and stripped and compressed, so it's possible your tools are misreporting, or I did something I wasn't expecting. At any rate, I haven't made that as a conscious design choice, and it's easy to change (fix?) compilation styles in a future build.

SQL engine: all from scratch

[+] vezzy-fnord|10 years ago|reply
The executable appears to be packed using vanilla UPX. Decompressing it and grepping the symbol table reveals the presence /lib/libc.so.6, /lib/libm.so.6 and /lib/libpthread.so.0.

By method of exclusion, it cannot be musl because musl unifies everything into a single /lib/libc.so.

It most likely is not uClibc because uClibc is typically symlinked to /lib/libc.so.0 from /lib/libuClibc.

Considering that the soname of /lib/libc.so.6 has been reserved by glibc since the 2.x days (to differentiate it from the Linux libc days of libc4 and libc5), it is most likely indeed glibc in use.

It's obvious this program isn't making use of NSS or anything like that, so it's doable to use glibc for static linking, if frustrating.

[+] siganakis|10 years ago|reply
I am really interested in this as it seems like a more accessible version of KDB - especially in its support for SQL.

I'd like to look into it further, but I can't find any information about licensing. Given that there is no source code in the repo, it appears that this isn't an open source project.

[+] e12e|10 years ago|reply
I would generally consider it a bug to not have any form of license in the repo, but from the top of the Readme (last modified 10 days ago):

"Contact Kevin (e.g., licensing, feature/documentation requests): [email protected]"

So if you're interested, I suggest you send Kevin an email.

Personally I think it would've been interesting if it was Free Software -- as this doesn't come with any license, it's essentially less useful than the 32bit kdb[1] -- and I can't imagine it's as feature complete, or has a similar level of documentation, support or real-world testing.

If the intention is to make the binaries deployable/usable, a license file/note in the Readme would be the absolute minimum requirement IMNHO.

Also find it strange to host just binary artefacts on github -- I can't see how that's useful for the author or the users. I suppose one could submit pull-requests against the python api, but then it would make more sense to split the repo in two -- one for the source-available (unknown license) python code -- and host the binary artefacts somewhere.

[1] http://kx.com/software-download.php

[+] mkevac|10 years ago|reply
What is tick?
[+] ddeck|10 years ago|reply
For the curious, in financial markets ticks are datapoints that represent every order (price, size, conditions etc.) placed (or modified) in the market for a particular security by a buyer or seller, as well as trades. In liquid equity markets across a large number of securities, this data can have a very high frequency and typical databases are not well suited for it's storage and retrieval.

More generally, tick databases are databases specialized for storing and querying high frequency time series data.

[+] napkindrawing|10 years ago|reply
As a data nerd, I got very excited once I read all the features, but I'm just assuming this will wither and die since it's closed-source =(
[+] james2vegas|10 years ago|reply
I see Linux and OS X, where are the iOS, Android and *BSD binaries?
[+] quotemstr|10 years ago|reply
No source code? I'm not even going to give this thing a glance. There is nothing so compelling in the database world that I would even be tempted to give up source access. What an anachronistic throwback this kind of thing is.
[+] quotemstr|10 years ago|reply
And it's packed using UPX too, presumably to "protect" the program somehow. Just say no to proprietary software from small vendors.