Stateless ZIP library – SLZ

[JoshTriplett]

Every DEFLATE compression block can either supply its own Huffman table or compress using a fixed table. You can emit a DEFLATE-compatible compression stream without ever emitting a custom Huffman table or depending on previous data. And you'll still do better than nothing for many types of data.

Given this, I wonder why HPACK invented a new compression scheme rather than just carefully using DEFLATE.

[andrewf]

The HPACK spec [1] remarks on this. Google carefully used DEFLATE in earlier SPDY versions [2] but HTTP/2 decided to be less clever.

Interestingly, SLZ's page comments on lack of history as a way to avoid CRIME attacks, which is one of the things the HTTP/2 people worried about. That seems like something it'd be easy for an implementation to screw up though (just add some buffering somewhere before the compressor...)

[1] https://http2.github.io/http2-spec/compression.html#rfc.sect... [2] https://www.chromium.org/spdy/spdy-protocol/spdy-protocol-dr...

[doublerebel]

Looks like SLZ hit 1.0 back in April [1]. Has anyone used it in production? Are packaged binaries available to get it up and running quickly?

[1]: http://git.1wt.eu/web/libslz.git/

[donatj]

Very interesting. This appears to do much better something I've been working on a PHP wrapper to do for a while now. I ended up currently opting out of compression as a whole and just using STORE zips to send multiple files at once without having to keep data in memory. This is absolutely worth looking into. It's been a long time since something on HN has gotten me this excited.

[striking]

Absolutely incredible. I had no idea this was even possible. Defeating CRIME while keeping compression? I can hardly believe it.

[amluto]

AFAICT this only avoids CRIME if there's never both attacker-controlled data and a secret in the same SLZ compress call.

[trippy_biscuits]

How does it address quines?

http://research.swtch.com/zip

[X-Cubed]

It's just a compressor, it doesn't provide any decompression support.

[JadeNB]

I love the fact that there is a ZIP quine, too, and I'll admit I immediately thought of that, but it's not clear to me what responsibility the library has to, or what the expectation on the user's part there is that it will, 'address' quines in any particular way.

[hinkley]

So would this be useful for compressed virtual memory? It only does compress so it would only halve the memory usage but that's something.

[andrewf]

If you're a webserver and want to serve clients which support gzip encoding, you need to be DEFLATE compatible.

For something like virtual memory where you'd control both the compressor and decompressor, there's little point in clever DEFLATE-compatible compressors. Instead, just pick any compressor which has the compression/CPU tradeoff you want.

[logicallee]

Ah yes, for that elusive "malloc-safe" spec requirement.

[tcas]

Having a fast malloc less compression library is extremely useful in resource constrained embedded systems world, where either memory usage should be minimized (you can still run modern Linux kernels on systems with 32MB of RAM, probably less), or where traditional memory management isn't available, such as a microcontroller.

In the microcontroller case you can get a 32bit ARM Cortex M0 for $2.00 qty 1 these days, allowing the bit-twiddling you find in compression libraries to compile successfully. Having a fast malloc-safe compression library is extremely useful. lz4 is an example of one which you can run on a microcontroller successfully, but isn't as compatible as zlib is.

[warmwaffles]

May not be useful to me, but it is still cool either way.

[ladon86]

How would one use this with HAProxy?