Shrink Go binaries 7x with this one weird trick

[shirro]

30MB binaries can be a bit of a problem for some people.

I started developing a system from home and when deploying to a VPS for testing and production. Transferring Go binaries was like waiting for a floppy disk to copy due to poor Internet infrastructure. Compression didn't help much.

It turned out to be significantly faster to push to a remote git and have a hook do the build. I think before you look to compress a go binary consider if there aren't other options.

[voltagex_]

30MB is on the small side. I've seen Electron apps go above 100 megabytes, and Android Studio is about 1.1 gigabytes. I feel sorry for anyone developing on less than 10 megabit down/up.

I've been throwing around an idea for a site called "Church of the Byte Savers" or some other silly name based around saving hard drive space and bandwidth. e.g. Dropbox is good because it has LAN sync, whereas OneDrive will send your data around the world just to get it to a second PC on your network.

[dietrichepp]

I feel your pain. My home internet connection is 24 Mbit/s down, but only 1 Mbit/s up. I asked my ISP for more, but it's basically impossible to get.

Thank goodness for multipart uploads, at least.

[s_kilk]

A few years back I thought it would be fun to do a Scala project (which ended up being http://nightchamber.com) while staying with my in-laws over christmas.

It got much less fun when each deploy to the prod server took an hour to complete.

[self]

Have you tried rsync? If most of your changes are small ones, it should help a lot.

    rsync --inplace -vaz sourcefile user@remote:/targetdir/

[riobard]

SSH compression is your friend.

[userbinator]

Note: I don't actually believe a 30MB static binary is a problem in this day and age

Depends what that 30MB does, or how much of it is actually needed - it's less of a problem if all those bytes are used in some form or another, than if a significant portion of them are "dead code" that is never used and only serves to take up space in memory, on disk, and in communications channels whenever it's transferred somewhere.

and I would not trade (build time | complexity | performance | debug-ability) for it

On the contrary, smaller binaries are usually going to be better at all of those, except perhaps "debug-ability" if you keep the debugging symbols in the same file.

[dozzie]

>> Note: I don't actually believe a 30MB static binary is a problem in this day and age

> Depends what that 30MB does, or how much of it is actually needed [....]

Moreover, let's consider a CLI tool suite. Dozen little tools, each doing perfectly what it was designed to (think of coreutils, util-linux, sysstat, binutils, or iproute2). Such a suite suddenly weighs 300MB, more than X.Org or PostgreSQL (or the two combined). Does it really do more than any of these two to justify the size? I doubt it.

[codedokode]

Note that there are downsides with this method:

- the startup time increases

- if you run several instances of an app, they consume more memory because code uses anon mapping instead of file mapping

- the whole executable file is loaded to memory at startup

[thrownaway2424]

"whole executable file is loaded to memory at startup" might be something you were going to do anyway, if you are running a latency-sensitive service. There's nothing worse than having to page in parts of the program after it starts serving.

[modeless]

A bigger downside: various virus scanners flag UPX-packed executables as suspicious.

[lossolo]

Do NOT strip Go binary:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717172

[FiloSottile]

Even if recent discussion is more mixed [0], that is why I linked to [1], recommended not to use strip(1), and explained -ldflags="-s -w" instead :)

[0] https://groups.google.com/d/msg/golang-dev/ABppMOjYP6w/6JsfT... [1] https://github.com/docker/docker/blob/2a95488f7843a773de2b54...

[userbinator]

Fixed in version golang/2:1.1.1-4

That bug is 3 years old and seems to be architecture-specific.

[mordocai]

Just be clear here: This is saying don't strip using the standard strip utility. As the author says in their post, go has flags to do stripping for go binaries.

[chetanahuja]

Note: I don't actually believe a 30MB static binary is a problem in this day and age

If you expand your context from "day and age" to "country, bandwidth provider and pricing" you might start believing the problem a bit more

[santaclaus]

> I don't actually believe a 30MB static binary is a problem in this day and age

Well, binary size is pretty important for embedded work. Or mobile work.

[jonathonf]

This "one weird trick" of stripping the binary, then compressing with goupx.

[prymitive]

Hard disk drive manufacturers hate him

[FiloSottile]

First paragraph:

> Ok, I lied, there's no weird trick. However, you can easily reduce a Go binary size by more than 6 times with some flags and common tools.

[elcapitan]

Waiting for the rebuttal "ENLARGE YOUR BINARY WITH THIS SIMPLE PILL".

[sov]

"With this simple .PLL!"

[skoczymroczny]

Admins hate him!

[thejay]

Op discovered exe packer.

[rhexs]

He's a full stack hacker, from the browser all the way down to ruby.

[sofaofthedamned]

Doesn't upx mess with same page merging though?

[FiloSottile]

As long as you don't mind having the entire uncompressed binary in memory, I don't think I see how. Can you elaborate?

[jedisct1]

Problem is that if you distribute UPX packed binaries, dumb antivirus will all raise an alarm.

[floatboth]

Also useful for Haskell: https://www.fpcomplete.com/blog/2015/05/haskell-web-server-i...

[donatj]

Haha, the author linked to my poorly written poorly conceived old grumbling post. I'm honored.

I should update the post as the binaries come out especially larger in the new versions of Go written in Go. I believe Hello Worldis up to at least 3mb.

[kevincox]

Isn't the output supposed to be identical between the C and Go versions?

[pklausler]

Not actually specific to golang.

[fibo]

Bravo Filippo!

[themartorana]

Honest question - mathematically, how can something be 7x smaller if x is not pre-defined? I always find it weird that anything can be more that 100% less than anything else because, you know, zero.

Am I way, way off base? Should I perhaps retake high school?

Edit: the smaller number is assumed 1x. Got it. Still feels weird.

[FiloSottile]

You might be overthinking it a bit :) I always see it the X used as the multiplication symbol, meaning "seven times smaller". You know, like when in games you get a 2x multiplier. I doubt that is the algebra "X" ;)

[michaelcampbell]

You're right, but as another poster noted "2x smaller" generally means 1/2 the original size. That usage, to me, is a horrible misuse of English, and is a pet peeve.

You also see it with slower; "the unoptimized code is 2x slower". Huh? Do you mean "takes 2x the time"?

I cringe every time.

[monochromatic]

It's not well-defined nomenclature. Generally what people mean when they say 7x less is 1/7 as much.

[seeing]

If a file is 100kb and is compressed to 50kb, that's 2x smaller. 50kb is still over zero.

[bobwaycott]

This is the second time in about a week that a post on HN has included a variation of "this weird trick" (the other being jlongster's article dealing with React) in the title. I can't help but parse these articles as equivalent to all the horrible ad spam all over the web about "this one weird trick" that will flatten your belly, tone your butt, help you make millions, give you a bigger penis, eradicate your cellulite, and a host of other "improvements" to your life.

Let's maybe spend a couple extra minutes coming up with less link-baity titles.

[ascendantlogic]

It's fascinating to watch you be so literal that the subtle mocking of those exact things has gone completely over your head.

[foota]

I think the title is a parody of those titles.