How are zlib, gzip and zip related?

[ctur]

What a great historical summary. Compression has moved on now but having grown up marveling at PKZip and maximizing usable space on very early computers, as well as compression in modems (v42bis ftw!), this field has always seemed magical.

These days it generally is better to prefer Zstandard to zlib/gzip for many reasons. And if you need seekable format, consider squashfs as a reasonable choice. These stand on the shoulders of the giants of zlib and zip but do indeed stand much higher in the modern world.

[michaelrpeskin]

I had forgotten about modem compression. Back in the BBS days when you had to upload files to get new files, you usually had a ratio (20 bytes download for every byte you uploaded). I would always use the PKZIP no compression option for the archive to upload because Z-Modem would take care of compression over the wire. So I didn't burn my daily time limit by uploading a large file and I got more credit for my download ratios.

I was a silly kid.

[lxgr]

> These days it generally is better to prefer Zstandard to zlib/gzip for many reasons.

I'd agree for new applications, but just like MP3, .gz files (and by extension .tar.gz/.tgz) and zlib streams will probably be around for a long time for compatibility reasons.

[pvorb]

I think zlib/gzip still has its place these days. It's still a decent choice for most use cases. If you don't know what usage patterns your program will see, zlib still might be a good choice. Plus, it's supported virtually everywhere, which makes it interesting for long-term storage. Often, using one of the modern alternatives is not worth the hassle.

[emmelaich]

Fun fact: in a sense. gzip can have multiple files, but not in a specially useful way ...

    $ echo meow >cat                                                            
    $ echo woof > dog                                                           
    $ gzip cat                                                                  
    $ gzip dog                                                                  
    $ cat cat.gz dog.gz >animals.gz                                             
    $ gunzip animals.gz                                                         
    $ cat animals                                                               
    meow                                                                        
    woof

[koolba]

> ... but not in a specially useful way ...

It can be very useful: https://github.com/google/crfs#introducing-stargz

[ericpauley]

Imo all file formats should be concatenable when possible. Thankfully ZStandard purposefully also supports this, which is a huge boon for combining files.

Fun fact, tar-files are also (semi-) concatenable, you'll just need to `-i` when decompressing. This also means compressed (using gz/zstd) tarfiles are also (semi-)concatenable!

[billyhoffman]

WARC files (used by the Internet Archive to power the Wayback machine, among others) use this trick too to have a a compressed file format that is seek-able to individual HTTP request/response records

[lxgr]

Wow, that's surprising (at least to me)!

Is there a limit in the default gunzip implementation? I'm aware of the concept of ZIP/tar bombs, but I wouldn't have expected gunzip to ever produce more than one output file, at least when invoked without options.

[cout]

Interesting -- I did not realize that the zip format supports lzma, bzip2, and zstd. What software supports those compression methods? Can Windows Explorer read zip files produced with those compression methods?

(I have been using 7zip for about 15 years to produce archive files that have an index and can quickly extract a single file and can use multiple cores for compression, but I would love to have an alternative, if one exists).

[ForkMeOnTinder]

7zip has a dropdown called "Compression method" in the "Add to Archive" dialog that lets you choose.

[pixl97]

Until windows 11, no, windows zip only seems to deal with COMPRESS/DEFLATE zip files.

[melagonster]

For people who first read this: the sweet part is in the comments :)

[dcow]

What’s even more sad is that the SO community has since consequently destroyed SO as the home for this type of info. This post would now be considered off topic as it’s “not a good format for a Q&A site”. You’d never see it happen today. Truly sad.

[miyuru]

His stackexchange profile is a gold mine itslef.

https://stackexchange.com/users/1136690/mark-adler#top-answe...

[stavros]

Hah, imagine asking Mark Adler for gzip history references.

[dustypotato]

Found this hilarious:

> This post is packed with so much history and information that I feel like some citations need be added

> I am the reference

(extracted a part of the conversation)

[tyingq]

Maybe a spoiler, but the "I" in "I am the reference" is Mark Adler:

https://en.wikipedia.org/wiki/Mark_Adler

[gmgmgmgmgm]

That's disallowed on Wikipedia. There, you must reference some "source". That "source" doesn't need to be reliable or correct, it just needs to be some random website that's not the actual person. First sources are disallowed.

[whalesalad]

Reminds me of when I was inadvertently arguing here on HN with the inventor of the actor model about what actors are

[FartyMcFarter]

"I'm the one who knocks".

[HexDecOctBin]

Is there an archive format that supports appending diff's of an existing file, so that multiple versions of the same file are stored? PKZIP has a proprietary extension (supposedly), but I couldn't find any open version of that.

(I was thinking of a creating a version control system whose .git directory equivalent is basically an archive file that can easily be emailed, etc.)

[pizza]

New versions of zstd allow you to produce patches using the trained dictionary feature

[kissgyorgy]

If you are interested in implementation details, how to unpack/decompress them, check out these Python implementations:

- https://github.com/onekey-sec/unblob/blob/main/unblob/handle...

- https://github.com/onekey-sec/unblob/blob/main/unblob/handle...

- https://github.com/onekey-sec/unblob/blob/main/unblob/handle...

[wiredfool]

The real question is: how are zlib and libz related?

[o11c]

zlib is the name of the project. libz is an implementation-detail name of the library on Unix-like systems.

[exposition]

There's also pzip/punzip (https://github.com/ybirader) for those wanting more performant (concurrent) zip/unzip.

Disclaimer: I'm the author.

[Dwedit]

See a highly upvoted answer in a question about zlib related things, suspect it was probably posted by Mark Adler, and turn out to be correct.

[raggi]

The answer is good, but is missing a key section:

Salty form: They're all quite slow compared to modern competitors.

[levzettelin]

What are some of those modern competitors?

[readyplayernull]

gzip can be used to (de)compress directories recursively in a variable:

FOO=$(tar cf - folderToCompress | gzip | base64)

echo $FOO | base64 - d | zcat | tar xf -

[encom]

(2013)