I have worked extensively with Bazel in the context of migrating a very large Java/Scala codebase from Gradle to Bazel.
My impression is that it is a first-class build system _specifically for Java and C++_. There are specific properties of the compilation and packaging ecosystem around those languages and runtimes that make them uniquely in need of tools like Bazel. This is not true for Go, where build speed and large codebases were a consideration up-front (especially with the new build caching features that shipped in Go 1.10), and are almost completely irrelevant in the context of "interpreted" languages like Python, Ruby or JavaScript.
There are plenty of other languages and runtimes that stand to benefit from a tool like Bazel, but just because Bazel is a fantastic tool in some contexts doesn't mean it brings that much to the table in others.
Actually it is very relevant for JavaScript at Google, given Closure Compiler. Most of our JS apps are type checked, code split and optimized and Bazel is how we structure the module dependencies. The Bazel build rules define independent logical modules, and build processes prune unused dependencies, and calculate a dependency graph which is used to organize an optimal module structure, this is then used by Closure Compiler’s CrossModuleCodeMotion to further reduce size by moving methods and properties down to later modules.
Granted, JS isn’t incrementally built, but the dependency graph pruning (removing unused files from the build that aren’t imported or referenced in JS code) is parallelizable.
I can speak for Java only. Most of my Java project, which are using Spring Boot and lots of libraries, gradle (previous is maven) works just fine. If there's any build issue, assume that it's not compilation issue, I just delete the `build/` and `.gradle` folders, then `./gradlew clean build` again.
I've took a hard look as bazel, for what they claim to be fast, correct, but the complicated setup and document just put me off.
Why should we consider bazel over gradle or maven for Java, or any other JVM languages?
It seems like Bazel's support for Go is still not first class and a lot of the issues mentioned here come from that.
But it strikes me as odd that they are figuring all of this out _now_, as opposed to a year ago when they first switched. Why did they switch to bazel in the first place? Did they do a trade study/test drive?
All of the problems mentioned (more complex build files and workflow, increased build times, lacking support for Go) you can find out right away as soon as you do a test drive. Unlike C++ or Java, it seems like the native go build tooling was always superior to what Bazel could offer, and they knew that. Why switch then?
In my experience using Bazel in a large and complex C++ codebase, it has been nothing short of amazing, compared to the CMake horror show that we had... We did have problems with it though, but mostly the lacking documentation in some areas, like building toolchain files, or the fact that support for Windows only got really good a couple of releases ago only. We did identify all of these in the initial migration effort and none of them was by itself a showstopper.
To give a tool a fair chance I think it has to be used daily to really grok the pleasure / pain. We were aware of some of the problems, but perhaps overly confident about managing them, and underestimated the frustration and lack of agency. Possibly we let it embed too far before we decided we have tried as hard as could be expected. Around 6 months in, once a number of us formally proposed that it be reverted, we took the bold decision that we should persevere no further. Having worked in C codebases in the past - I can imagine it is almost a miraculous improvement there.
We've been using Bazel at work, where we have a monorepo with lots of different languages at play (Python, Go, Java, Scala, JavaScript, CoffeeScript, TypeScript, Protobuf, just to name the major players) and I like that it makes all of them use a consistent interface for configuring and running builds. It helps that we have a specific team handling most of the Bazel work - and it costs about 10% of all of that team's time.
That's about the only thing I like about it, though. It's the most complicated build system I've ever used and I still don't fully grok it after 7 months working in this position.
> where we have a monorepo with lots of different languages
isn't that the only reason to use something like bazel?
I considered using it for our go, java, scala, typescript and c# project, but I would've never used if for go only.
If your entire codebase is Go, you really shouldn't be using Bazel/Pants/Buck/Please. Go's built-in build system is perfectly adequate for just Go, with light tooling for generated code (eg. protobufs).
It's when you have multiple languages and want to be able to build/test only what has changed that you need a Blaze-alike.
With regards to Gazelle: is it intended to be run every build, on every file? Or is it intended to be hooked into your editor, and run on save to update your BUILD files as you go? At Google, I used the latter kind of tool, and it worked just fine. If it missed something, you'd notice when the CI system failed to build/test your change.
Regarding Gazelle, yes, the CI would break early if it detected a none porcelain repo after running Gazelle itself. I think others may have configured their editors, but for me the occasional 30+s run time (and as mentioned in some workflows 3-4 mins) on file save - plus the already creaking VSCode plugins was too much to bear.
Have you tried Nix? It's like Bazel, but without incremental builds. The boons are pretty great.
I'm extremely disappointed, as a career SRE, to see "hope is not a strategy" trampled in the mud. I understand that perhaps the author has succumbed to epistemic helplessness and given up on having computers work without optimism or belief, but computers do not operate on any ruleset which includes hope, and we should not encourage magical thinking.
Not trampling, just holding at arms length. Recognising the reality of the economics of most projects. Unless you live in a completely deterministic world, devoid of human fallibility, or perhaps omnipotent, or simply have unlimited time or resource, at some point you are going to have to admit you just don't know, and you are managing the percentages. Delusional overconfidence is more close to magical thinking than recognising the reality that complex systems will fail in surprising ways, and that it costs ever increasing resource to reduce the risks with ever diminishing returns, until you are forced to stop. The ruleset the computer abides by probably represents a fraction of the factors affecting success. If you feel you have never got to a point where there is an element of faith involved in your choice, then Im envious.
> Have you tried Nix? It's like Bazel, but without incremental builds.
Well, depending on how granular you expect your incremental builds to be, it can indeed have "incremental builds". Any build step that produces a derivation that ends up in the nix store will be lazily evaluated and not regenerated if its input parameters haven't changed. You could go down the route of outputting each compiled object file as a derivation...
When we used Bazel at Staffjoy, we had two main issues:
(1) NPM dependency management was hard. We ended up committing built dist files, like index.html
(2) We ended up duplicating dependencies between Bazel and Glide because tools like goimports and linters could not read the Bazel remote dependencies system.
It added a dependency on the JVM. (I probably have an overly painful past with the JVM in production), suffice to say we had to bump up the RAM on the build machines.
This alone is a reason not to use Bazel. It's designed for the enterprise in mind, where JVMs are common, sharing a common build environment in a team is easy (or at least, the setup time isn't an issue).
For a community project, this type of thing wouldn't fly. There's a reason Meson is winning the latest build-systems war: it only depends on python3 & ninja.
> There's a reason Meson is winning the latest build-systems war:
What do you mean by winning? I haven't heard of Meson until the last few weeks, and I haven't seen it consumed by any open source projects on GitHub, although my search biases more towards JVM things anyways.
> it only depends on python3 & ninja
I don't see how depending on both Python3 and Ninja is any less dependencies than the JVM. Python3 isn't really available by default on most distributions (maybe that is changing soon), so there is still the overhead of installing it, right? Public CI systems like Travis CI and Circle CI have images that make having a JDK easy.
> Hope is a strategy [...] At some point we have to hope and assume. for example eventually we hope the compiler authors did a good job with the next version we are about to use, or we assume that the kernel fix was good.
Or you can do a very slow, controlled rollout of the new version and see what happens. With all the systems I worked with while at Google, both as a SRE and a SWE, whenever we had a new version to release, we'd update one task in one cluster, let it run for a day or two, check the logs and the metrics... if it was OK, update one job in one cluster, then repeat the process with an entire cluster (the designated canary cluster), and only then release to the rest of the clusters. If any of these went wrong, we'd rollback or patch, depending on the severity. We rarely missed our weekly releases.
I'm sure the same applies to a new compiler version or a new kernel fix. You don't need to assume anything.
hmm, yes, this was always going to be a contentious point, and highly subjective. I think the main issue here is it was in a large part also a very human impact, which is much harder to measure. I think it was fair to say that our build system at the time was more rigid and testing fully both flows not within our grasp. I discussed this point about 'hope' with a colleague, I think I agree with his conclusion: "Google actually designs some of the chips it uses! And runs its own power stations - I think if anyone could legitimately say "Hope is not a Strategy" it might be them."
Thanks for the comments on Bazel -- I am evaluating a proper build system for us right now for a data science codebase almost entirely in Python, with a lot of scientific computing dependencies. Bazel doesn't yet seem to have first-class support for Python, so that's out, and I'm running into a lot of problems with Twitter Pants, although their Slack channel is really helpful.
Did you ever try using Pants or Buck, or a more barebones approach like Ninja?
P.S.: An explanation of the pun on "Basil Fawlty" (as a non-Brit, I thought, what on earth is that?) in the article might be helpful.
Also, go already has a very good build sysmtem build-in, and hazel really shines when:
- you have a complex codebase with multi-languages: one can build a tool in one language and use it as tool for a another language.
- you simply have a really large code base - you can work on the path //service1/my/service and your colleague can work on //service2/their/service, and only the path got changed needs to rebuild every time.
It eschews docker, and Dockerfiles and their non-determinism, and builds the layers itself, and uses a python lib [1] to push and pull container images.
In order to ensure reproducibility/determinism, however, Bazel doesn't have an equivalent of the RUN instruction. You have to use other Bazel rules to fetch and produce artefacts and add them to an image, and there aren't always rules for what it is you want to do (I have a spot of bother with installing pip packages, for which there is apparently alpha support).
This is, I think, the thing with Bazel: it has to re-invent everything to ensure this reproducibility and hermetic seal, because it doesn't trust existing build tools to do this, and quite rightly so if this is what you're seeking. But I suspect it's going to be painful doing things outside the mainstream supported stuff.
What's the context for this - as in, the 'we' in question and the thing they were building with bazel? And are they really posting from March 16, 2018?
I probably should have made the context clearer earlier in the post - later on I mention :
"To set the scene, our codebase is mainly Go, including vendor directories, and a considerable amount of data compiled in, we have 2.5M lines of code, a full build from a clean clone on one of our Jenkins slaves takes 1 minute.
We had between 8 and 20 engineers working on that codebase. Importantly we all develop on Mac, but run in production in linux."
Interesting to have feedback on Bazel usage outside of Google. Bazel looks like a great tool, but it's good to see the community starting to explore its sweet spots and drawbacks.
[+] [-] venantius|8 years ago|reply
My impression is that it is a first-class build system _specifically for Java and C++_. There are specific properties of the compilation and packaging ecosystem around those languages and runtimes that make them uniquely in need of tools like Bazel. This is not true for Go, where build speed and large codebases were a consideration up-front (especially with the new build caching features that shipped in Go 1.10), and are almost completely irrelevant in the context of "interpreted" languages like Python, Ruby or JavaScript.
There are plenty of other languages and runtimes that stand to benefit from a tool like Bazel, but just because Bazel is a fantastic tool in some contexts doesn't mean it brings that much to the table in others.
[+] [-] cromwellian|8 years ago|reply
Granted, JS isn’t incrementally built, but the dependency graph pruning (removing unused files from the build that aren’t imported or referenced in JS code) is parallelizable.
[+] [-] letientai299|8 years ago|reply
I've took a hard look as bazel, for what they claim to be fast, correct, but the complicated setup and document just put me off.
Why should we consider bazel over gradle or maven for Java, or any other JVM languages?
[+] [-] danmux|8 years ago|reply
[+] [-] elteto|8 years ago|reply
But it strikes me as odd that they are figuring all of this out _now_, as opposed to a year ago when they first switched. Why did they switch to bazel in the first place? Did they do a trade study/test drive?
All of the problems mentioned (more complex build files and workflow, increased build times, lacking support for Go) you can find out right away as soon as you do a test drive. Unlike C++ or Java, it seems like the native go build tooling was always superior to what Bazel could offer, and they knew that. Why switch then?
In my experience using Bazel in a large and complex C++ codebase, it has been nothing short of amazing, compared to the CMake horror show that we had... We did have problems with it though, but mostly the lacking documentation in some areas, like building toolchain files, or the fact that support for Windows only got really good a couple of releases ago only. We did identify all of these in the initial migration effort and none of them was by itself a showstopper.
[+] [-] danmux|8 years ago|reply
[+] [-] Sir_Cmpwn|8 years ago|reply
That's about the only thing I like about it, though. It's the most complicated build system I've ever used and I still don't fully grok it after 7 months working in this position.
[+] [-] merb|8 years ago|reply
isn't that the only reason to use something like bazel? I considered using it for our go, java, scala, typescript and c# project, but I would've never used if for go only.
[+] [-] zellyn|8 years ago|reply
It's when you have multiple languages and want to be able to build/test only what has changed that you need a Blaze-alike.
With regards to Gazelle: is it intended to be run every build, on every file? Or is it intended to be hooked into your editor, and run on save to update your BUILD files as you go? At Google, I used the latter kind of tool, and it worked just fine. If it missed something, you'd notice when the CI system failed to build/test your change.
[+] [-] danmux|8 years ago|reply
[+] [-] EtDybNuvCu|8 years ago|reply
I'm extremely disappointed, as a career SRE, to see "hope is not a strategy" trampled in the mud. I understand that perhaps the author has succumbed to epistemic helplessness and given up on having computers work without optimism or belief, but computers do not operate on any ruleset which includes hope, and we should not encourage magical thinking.
[+] [-] danmux|8 years ago|reply
[+] [-] ris|8 years ago|reply
Well, depending on how granular you expect your incremental builds to be, it can indeed have "incremental builds". Any build step that produces a derivation that ends up in the nix store will be lazily evaluated and not regenerated if its input parameters haven't changed. You could go down the route of outputting each compiled object file as a derivation...
[+] [-] philip1209|8 years ago|reply
(1) NPM dependency management was hard. We ended up committing built dist files, like index.html
(2) We ended up duplicating dependencies between Bazel and Glide because tools like goimports and linters could not read the Bazel remote dependencies system.
We open-sourced the repo with Bazel when we shut down -> http://github.com/staffjoy/v2
[+] [-] Aissen|8 years ago|reply
This alone is a reason not to use Bazel. It's designed for the enterprise in mind, where JVMs are common, sharing a common build environment in a team is easy (or at least, the setup time isn't an issue).
For a community project, this type of thing wouldn't fly. There's a reason Meson is winning the latest build-systems war: it only depends on python3 & ninja.
[+] [-] mkobit|8 years ago|reply
What do you mean by winning? I haven't heard of Meson until the last few weeks, and I haven't seen it consumed by any open source projects on GitHub, although my search biases more towards JVM things anyways.
> it only depends on python3 & ninja
I don't see how depending on both Python3 and Ninja is any less dependencies than the JVM. Python3 isn't really available by default on most distributions (maybe that is changing soon), so there is still the overhead of installing it, right? Public CI systems like Travis CI and Circle CI have images that make having a JDK easy.
[+] [-] ggambetta|8 years ago|reply
Or you can do a very slow, controlled rollout of the new version and see what happens. With all the systems I worked with while at Google, both as a SRE and a SWE, whenever we had a new version to release, we'd update one task in one cluster, let it run for a day or two, check the logs and the metrics... if it was OK, update one job in one cluster, then repeat the process with an entire cluster (the designated canary cluster), and only then release to the rest of the clusters. If any of these went wrong, we'd rollback or patch, depending on the severity. We rarely missed our weekly releases.
I'm sure the same applies to a new compiler version or a new kernel fix. You don't need to assume anything.
[+] [-] danmux|8 years ago|reply
[+] [-] sarabande|8 years ago|reply
Did you ever try using Pants or Buck, or a more barebones approach like Ninja?
P.S.: An explanation of the pun on "Basil Fawlty" (as a non-Brit, I thought, what on earth is that?) in the article might be helpful.
[+] [-] philbarr|8 years ago|reply
https://www.youtube.com/watch?v=mv0onXhyLlE
Imagine the car is your build tool...
[+] [-] humanrebar|8 years ago|reply
"The plots centre on tense, rude and put-upon owner Basil Fawlty (Cleese)..."
https://en.wikipedia.org/wiki/Fawlty_Towers
[+] [-] davidstanke|8 years ago|reply
[+] [-] nine_k|8 years ago|reply
[+] [-] danmux|8 years ago|reply
I have added a link to the classic scene as referenced by philbarr
[+] [-] danmux|8 years ago|reply
[+] [-] jasikpark|8 years ago|reply
[+] [-] erain|8 years ago|reply
You can checkout Bazel's docker rule: https://github.com/bazelbuild/rules_docker
Also, go already has a very good build sysmtem build-in, and hazel really shines when: - you have a complex codebase with multi-languages: one can build a tool in one language and use it as tool for a another language. - you simply have a really large code base - you can work on the path //service1/my/service and your colleague can work on //service2/their/service, and only the path got changed needs to rebuild every time.
[+] [-] leg100|8 years ago|reply
In order to ensure reproducibility/determinism, however, Bazel doesn't have an equivalent of the RUN instruction. You have to use other Bazel rules to fetch and produce artefacts and add them to an image, and there aren't always rules for what it is you want to do (I have a spot of bother with installing pip packages, for which there is apparently alpha support).
This is, I think, the thing with Bazel: it has to re-invent everything to ensure this reproducibility and hermetic seal, because it doesn't trust existing build tools to do this, and quite rightly so if this is what you're seeking. But I suspect it's going to be painful doing things outside the mainstream supported stuff.
[1] https://github.com/google/containerregistry
[+] [-] pvg|8 years ago|reply
[+] [-] danmux|8 years ago|reply
"To set the scene, our codebase is mainly Go, including vendor directories, and a considerable amount of data compiled in, we have 2.5M lines of code, a full build from a clean clone on one of our Jenkins slaves takes 1 minute.
We had between 8 and 20 engineers working on that codebase. Importantly we all develop on Mac, but run in production in linux."
[+] [-] ndh2|8 years ago|reply
[+] [-] meanmrmustard92|8 years ago|reply
[+] [-] kemenaran|8 years ago|reply