If a Build Takes 4 Hours, Run It Every 4 Hours

[_ph_]

I don't know what the author is building that a build is taking 4 hours, but I am aware that there are huge software projects with build times like that and much longer. I agree that getting quick feedback to changes is vital for quick overall development. This shows the values of development stacks which reduce the build times a lot. As a resultmore incremental development can be done. For example, the build times were one of the architectural goals when designing the Go language.

Dynamic languages are also a great tool to fight build times. Even if it is unsuitable for large projects to be entirely built on dynamic languages, a great technique is using a mix of static and dynamic. Build the core of the application in a static language, but more as a set of libraries to be called from the dynamic layer. The dynamic layer is where most of the application logic resides, easy to be changed. And parts which have stabilized reasonably can still be moved over to the static part.

[scarface74]

So by using a dynamic language, you trade the cost of a long build time with possible runtime errors that would have easily been caught by the build system? Instead of paying the cost once per build you are now inflicting runtime errors on your customers.

[castratikron]

I wrote physics simulation code some time ago. The build was fast, but since the simulation speed was about 10ns/day the tests would take several hours to run. And the only way to know if your change fixed something is if the tests passed.

So it took several hours to get feedback for a change. But it also took about that long to come up with a new idea to try. Eventually I had a pipeline running where I could fire off a few ideas throughout the day, keep track of the change in the build, and iterate that way.

[dpc_pw]

You're going to lose that gained time as soon as your "build" includes running tests. And you'll need more tests because in dynamic langueses generally nothing works until you check it with a test.

Mixing lanuages is a waste. It makes everything harder operationally, technically and business wise.

The solution to long builds is projects structure that supports caching and parallelism. No one ever changes more than couple thousands lines even in a multi million line project, so why would everything had to get rebuilt?

I've seen ci pipelines building millions of lines in under a minute, no sweat.

[mister_hn]

Have you ever tried to build Chromium or Firefox yourself? If you have a slow machine, the build can take even 10 hours or more

[noisy_boy]

We have Maven builds that complete in a few mins via Jenkins but I have been exploring if it can be done even faster. The builds are written using the Jenkins declarative pipeline with parallelization etc but feels like it is just not fast enough (the slowness is probably because of the whole lot of intermediate IO/back-and-forth between Jenkins master and slaves). Wish there was a build setup that was faster while being as feature rich as Jenkins while supporting Maven builds.

[mister_hn]

Try the flag -T XC where X is the number of threads you want (similar to make -jX)

[tedunangst]

Tying your build frequency to build time and then using that as motivation to decrease build time is probably the important bit here, but only if you include the second part.

[royosherove]

That's a good way of putting it. In TOC terms, the build time becomes the new constraint to solve, but only after you solve the build frequency constraint.

[teleforce]

If the long build time for static typing programming language is the concerns perhaps you can try have look at D programming language since it is optimized for fast compilation [1].

Around last year someone even managed to speed up its compilation time even further and it is also featured in Hacker News but cannot find the link now.

I think in the future most of the static typing programming languages will be mainly based on Single Static Assignment (SSA) technique then the compilation could really fly.

[1]https://www.drdobbs.com/cpp/increasing-compiler-speed-by-ove...

[2]https://www.cs.princeton.edu/~appel/papers/ssafun.pdf

[gregdoesit]

Alternatively, you can parallelise parts of your build (eg the tests, static analysis etc) and speed it up. Then, if you have more changes coming in than you could parallelise is when you can start to do some advanced things like trying to predict which changes might break the build.

There was a longer discussion on one of these approaches done by Uber[1] on HN. Granted it’s a lot of investment, it’s interesting to explore ways of dealing with long builds that need to run frequently.

https://news.ycombinator.com/item?id=19692820

[larrybud]

Or, just parallelize the entire build. eg, if you have four build servers, you could kick off a build every hour. Easy using VM's, trivial in the cloud.

[royosherove]

True, but that type of optimization is quite moot if you're only running the build at night (i.e, that's not the first constraint to solve). In the article I mention that first get the builds to be tight one after another, then work on build times because that becomes the constraint.

[DmitryOlshansky]

Why not every hour or even half-hour?

Provided that you can run multiple builds in parallel but each of them has somehow limited parallelization.

[tlarkworthy]

Why don't we run it on every change?

[eska]

It's a bandwith vs latency issue.

Imagine you make 5 commits in 4 hours. You now have 5 builds queued up and 20 hours of latency. So one would probably prefer to only build the last commit, and maybe queue the older commits up for later. But if you make another change then you'd need to pause the build of the older commit, or you'd be at up to 8 hours of latency instead of 4..

[bryanrasmussen]

If a build takes 4 hours it argues that a build on that system is a pretty expensive operation, unless changes are very infrequent, which in most systems is not the case, a build run on every change would seem to be highly problematic.

[royosherove]

That would be ideal, but many organizations don't yet have that capacity for large builds/deploys (yet).