Kraken: 3x faster decompression than zlib

[rb2k_]

Another interesting compression project is "Zstandard"

Open source, high performance, decent compression and a "dictionary mode" that's especially fun when it comes to compressing a lot of small data.

http://www.zstd.net

Previous discussion over at https://news.ycombinator.com/item?id=8941955

[eln1]

Also lzturbo is very interesting, closed source but you one can download exec: https://sites.google.com/site/powturbo/ https://github.com/powturbo/TurboBench

[richdougherty]

Here's a nice graph from the zstd site, showing the trade off between compression and speed for several different algorithms:

https://raw.githubusercontent.com/Cyan4973/zstd/master/image...

[Johnie]

There's too many projects named Kraken:

* http://krakenjs.com/ - Javascript framework

* https://www.kraken.com/ - BTC exchange

* https://kraken.io/ - Image Optimizer

[dwc]

There are too many projects of any name (most never see the light of day), but somehow all these Krakens get released.

[dikaiosune]

https://ccb.jhu.edu/software/kraken/ <- taxonomic ID assignment for metagenomic sequence samples

[Osiris]

Agreed.

Another: GitKraken - https://www.gitkraken.com

[jug]

Since this was a Kraken benchmark, I first thought it was some thing benchmarked in the Kraken benchmark.

http://krakenbenchmark.mozilla.org/

I should've known better...

[josteink]

I guess someone was ordered to release the Kraken(s), and as Eddie Izzard famously observed[1], nobody ever issued an order to retrieve the Kraken.

To the grumpy crowd: Sorry in adance for the reddit-like comment. I feel a bit light-spirited today :)

[1] https://www.youtube.com/watch?v=Cyq4zT0h0yk

[userbinator]

I guess it's yet another optimised LZ variant, and it's no surprise that LZ-based/inspired algorithms are fast, because their main loop is very simple and does not involve the intense bit manipulation of Huffman/Arithmetic algorithms. That's how very carefully optimised LZ can end up being faster than memcpy() in some cases:

http://www.oldskool.org/pc/lz4_8088

[mmozeiko]

Few benchmarks from @rygorous:

https://gist.github.com/rygorous/67c78fde615e078117ea1870472... (on PC)

https://gist.github.com/rygorous/9563760e24463a37984c117b572... (on PS4)

[vardump]

So slower than LZ4? I wonder how Kraken compression rate compares to LZ4.

[EvgeniyZh]

How is it compared to Zstandard https://github.com/Cyan4973/zstd ?

[wolf550e]

Especially with level 22 with --ultra, I found the compression to be good and decompression very fast. Can't wait for 1.0 for format freeze (currently decompressor contains code for previous versions, but that's a hack).

[muizelaar]

Unfortunately not open source.

[vvanders]

Feel free to downvote me but some people need to eat.

Game development doesn't really match well with the consultancy model of most commercial Open Source companies.

Also the RAD guys are awesome, they've been putting out solid tools for years.

[kazinator]

Is this 3x faster than zlib at decompressing on zlib-compressed data, or 3x faster on samples compressed with its own format versus the same samples compressed with zlib?

[mmozeiko]

It is custom compression algorithm. It has its own format.

[Jerry2]

Does anyone know how Apple's lzfse [0] compares to some of these other compression algos?

>Beginning with iOS 9 and OSX 10.11 El Capitan, we provide Apple’s proprietary compression algorithm, LZFSE. LZFSE is a new algorithm, matching the compression ratio of ZLIB level 5, but with much higher energy efficiency and speed (between 2x and 3x) for both encode and decode operations.

I'm interested because of the low energy characteristics...

[0] : https://developer.apple.com/library/tvos/documentation/Perfo...

[MichaelGG]

What about LZHAM; I thought that was the top "slow compress, good ratio, fast decompress" contender for games and such?

[panic]

This article has some comparisons between Oodle's LZNA and LZHAM (at least as of a year ago): http://cbloomrants.blogspot.com/2015/05/05-09-15-oodle-lzna....

[iso-8859-1]

It is no secret that vanilla zlib is slow. Would have been interesting if they compared with https://github.com/Dead2/zlib-ng

[noir_lord]

Is the speed difference algorithmic, implementation optimization or a mix of both?

[mistercow]

They say it's a mix of both.

[stellar2]

Looking at the figures, it seems to have roughly the same performance as Zstandard in high compression mode.

[Gratsby]

But what is it's Weissman score?

[gjkood]

I was just watching Silicon Valley yesterday and this is exactly what I was thinking when I read the title. 6.0+?