One issue we hit with our CI, and mix of build systems is this - given a changelist, find out which targets needs to be built, and which one needs to be tested on pre-submit, and which on post-submit.
With that, we end up paying so much extra time building everything over and over without need, and then not building things that we ought to.
So that's one reason to switch, but at the same time lots of people simply do not get it. To them it seems intrusive, new, opinionated, and makes them not happy to use it. I've used it for 2+ years at google, and yes initially - was WTF is this? Then it hit me... And I'm sure the same is for buck, pants, please.build, gn and other similar systems.
At the end of the day, you need way to express "end to end" your build graph, from any single individual source file, shell script, or configuration downto building your executables, deploying them, etc.
It's an industry tool, that needs to be looked, and if it takes 5 people to support it, then it takes 5 people to support it, but you won't be wasting other peeople's time on issues like - "Why this build in the CI did not trigger?", why it takes, and wastes my time (waiting for presubmit), etc.
Yes, it does not come for free, but it's worth knowing and trying it out at least.
If nothing else, here is the takeaway - Try to use a system with static graph, where relationships are known before you start building things. It's not always there, e.g. your #include "header.h" file is dynamic, but bazel forces you to express even that, and later it catches it whether you've done it, and breaks unless it's fixed.
> Then it hit me... And I'm sure the same is for buck, pants, please.build, gn and other similar systems.
There’s an exercise you can do where you design a build system on the basis that it shouldn't do unnecessary work (which can be very slow and frustrating in practice).
My personal experience is that you can really quickly get to the point where just reading the entire graph into memory gets expensive. People talk about how Google is huge… but long before you get to that scale, you can end up with a build graph that just takes forever to parse and evaluate. (At Google's scale, it doesn't even fit in memory any more.)
So you decide that, as a hard design requirement, you should be able to only load the portion of the repository that you are building. And then you want to make this cacheable, so you can change the repository and know what’s changed in some quick / reasonable way.
If you go down this path, you end up rediscovering some of the big design decisions behind Bazel, Buck, Pants, Please, and GN.
Good heuristic for whether it's worth considering moving to bazel for your build system:
- Do you have 200+ developers working on a monorepo?
- Are you willing to vendor all of your dependencies and maintain their builds yourself?
If so, consider it. The productivity you're losing to unnecessary rebuilding and re-running unchanged unit tests will probably be paid back if you can contort your development process to the one Bazel expects.
If you're a small shop, the benefits Bazel is going to provide over, say, Make (or whatever standard build system your primary language uses), are going to be minimal. And the overhead of maintaining Bazel is going to cost you a ton of developer time you may not be able to afford.
Another factor: are your languages supported by bazel? If you use the same languages that google uses (C++, Python, go), it's fair to say that those are well supported. For all other languages, even if they are widely used outside of Google (JavaScript, nodejs), you may be out of luck.
So I always wonder how Google does this. Somehow they're able to determine which individual unit test are impacted by a change. Maybe this is only for Java but I'm pretty sure I recall claims that they can change 1 line & know that only 1 other test case is impacted (i.e. not even the other test cases within the same unit). Now Google's monorepo does imply also that if you are changing a line that is foundationally part of a lot of other components that they get rebuilt & retested too - that's the risk tolerance Google has chosen (i.e. they'd rather tests take longer if you're changing a core component rather than skipping tests & risking destabilizing 1000+ other engineers).
I was subjected to bazel on a small project because the manager insisted we use it. The rest of the company used a number of either custom tools or cmake or premske.
It is utter hell when you have tons of third party libraries (internal or external to the company) that you don’t have the source to and it is especially painful when trying to integrate bazels behavior against other build systems. Also bazels packaging and use of internal symlink renaming was a constant source of suffering. Bazel pretty much destroys a number of totally valid work linux commands for looking for so files.
Bazel might be useful in the case of a monorepo with a massive engineering pool AND a massive cloud infrastructure backing that repo to handle all the artifact sharing, but after having used cmake, premake, waf, random perl and ruby scripts, or just checking vs projects into perforce manually, I’d pick any of those before bazel for most projects. I say that having worked on code bases from a few 10s of thousands to 25+ million LoC with teams small, large, and distributed.
Bazel probably has its place but I have yet to find it.
My personal experience is that Bazel cut through a bunch of the problems that I’ve had with CMake, Waf/SCons, etc. Builds were fragile, they were not reproducible, and there were implicit dependencies. This is mostly as someone who’s rewritten a few build systems, rather than as someone who’s been subjected to build systems by others (I mostly inflict these changes on other people). With Bazel, I have much higher confidence that I’ll get consistent results when I check out the repository on different computers or work with other people.
That said, the major sore point with Bazel for me is the general lack of expertise about how to work with it sanely. Depending on what part you’re looking at, it’s somehow both “too opinionated” and “too flexible” at the same time.
I think it will capture a big chunk of the mindshare for build systems over the next few years, and you’ll see more and more of it. Over that time, people will develop the expertise and best practices for different development problems.
For managing third-party dependencies specifically, Bazel gives you a ton of options, including options that only really make sense for huge orgs like Google. Google vendors their third-party libraries directly into the monorepo. If that doesn’t make sense for your org, Bazel lets you work with external Git repos, with artifact repositories, with package repositories like NPM, or with tools like pkg-config.
The thing that makes this hell, right now, is that few people how to use it well and the documentation is rough. I’m personally very happy with it, even for small codebases, but I’ve used it a lot.
For anyone thinking about Bazel for their project/organization... run as fast as you can in the opposite direction. It's easily the most complex and unintuitive build systems in the world, and I'm saying that as someone who used SCons. At the last job where I used it, I was on a team of 5 whose responsibilities included Bazel upkeep, which required anywhere from 10 to 50% of our time. This was used by a broader engineering team of 50, working on 3-5 "big" projects and a few dozen small ones.
If you are an organization with a large enough codebase (especially if it's in a monorepo) that you need a shared remote cache of build artifacts, or remote build sharding and execution, and have multiple languages (even protocol buffers) interacting in complex dependencies, then you should run as fast as you can away from less rigorous Blaze-alikes (Pants, Buck, etc.) straight towards Bazel.
Yes, it's complicated, but it's also quite rigorous, and the rigor pays off.
(We at Square had already found a Blaze-alike necessary. We are currently busy converting our Java build from Pants to Bazel.)
I've only experienced issues when using bazel with third party package management systems. If you can own all of your source it, and it's descendants, are easily my favorite build systems. It's features complement modern software development in a very ergonomic way: uniform build language, API for learning about your source, testing your entire code base in every language with one command, hermetic and reproducible builds, distributed builds, and caching I can actually trust.
Using Bazel with external packages on the other hand is one of the most tedious and frustrating endeavors imaginable. If you can vendor all of your source it's much less frustrating. This is extremely manageable in the C and C++ worlds where there aren't really any package mangers and you end up needing to do that anyway.
Pretty much the same experience. 2y old monorepo company with >150 engineers now, being slowed down by hacks upon hacks in the usage of the bazel build system with around 5 people at the company understanding how bazel works. And those people are discouraged quickly when trying to improve things because of the shear brittleness of the existing build files and "optionated" approaches in the community that don't quite cut our usecases. Even things like enabling remote caching and execution take many months (and even external companies) and still drag on.
Edit: I don't want to come off as too negative about Bazel. But I really think it needs more time and is nowhere near something that I would call a 1.0 let alone a 2.0
I’ve been using it for the last 2 years, and I am not doing any project again without it.
Bazel is not that complex if you start a project with it. Migrating to it and learning it at the same time will be hard though since you’re likely to uncover a lot of skeletons.
Seconded. Any org with less than a few 100 engineers (and many with that many and more) would do better to stay away from this. I've had the dubious honor of using it for one project and to me the slogan is 'tools should work, not require attention', rather than the opposite. Bazel will require a lot of your attention and for smaller companies that could easily be a big percentage of their available capacity.
For very large organizations with the capability of assigning one or more teams to tooling it may very well be the right choice.
Could you elaborate? I've been using it for a decade, for all my projects big and small, and it's been a _massive_ time saver compared to any of the alternatives. It's fast, well documented, and it doesn't rebuild/retest stuff when it doesn't have to. I've also done multiplatform builds with it, as well as cross-builds to ARM. Not once did I have the need for any "upkeep". At Google there was a team to do that, of course, but even outside Google, even very early on when Bazel was just released, the maintenance is pretty minimal, and the build files are by far the most readable of any build system I have used so far in 20+ years in this industry.
Hey Drew, I'm a huge fan and have a lot of respect for your work.
I feel like I see the pattern of people on HN being disappointed in some of the tools that come out of Google and other large engineering orgs, when they don't work out really well in orgs that are not operating at the same scale. People have similar complaints about the complexity of other projects that come out of Google. K8s comes to mind as one such example. Often times these tools must be robust to such a large variety of uses that they are simply overkill for smaller organizations. I'll readily admit that I could be wrong and Bazel is simply poorly designed, but it is perhaps worth considering that the build system used by an engineering team of 50 need not be as complex as the build system used by one of the largest engineering orgs in the world. My guess is we'd see a lot less backlash if people tried to step off the hype train for a moment and critically evaluate whether they really need to use something like Bazel or K8s when something simpler would suffice.
I was talking with an engineer who saw two people burn out over Bazel, though the specific gripe was with Scala support. I'd expect first-class languages at Google (C++, Java, Python, Go) to get better support.
I think it entirely depends on how you use tools. Where I currently work people decided that it would be a great idea to extend waf (another niche build system) with all kinds of features. If you ever worked with waf you can probably imagine how bad that can turn out. As long as you stick to bazel’s default features and don’t start to extend it, I’ve found it really pleasant to use especially for C++ and python projects (where you want to expose C++ libraries to python as well). If you start to extend it it can probably become horrible really quickly. The only experience I have with that are abortive attempts at integrating system verilog tools (turned out to be hard) and integrating a custom GCC toolchain (worked fairly well)
As with many projects using semantic versioning, the major version bump just signifies there are some breaking changes. Most projects will just switch from 1.x to 2 work noticing.
I can’t understand why the Bazel team hasn’t learnt from Go team how to handle breaking changes. Bazel is an amazing piece of tech, but it can definitely be a lot of work to keep up to date.
That "changed some corner cases that likely won't affect you" and "rewrite it all" looks the same in SemVer makes it next to useless, not that any other system would be better. We just shouldn't have any expectations about version numbers conveying much information.
Bazel 1.0 was released on October 10. Not much more than two months ago. At best, semantic versioning expresses the intent to break existing code. It's an anti-pattern. It's not a feature. Projects using semantic versioning are explicitly caveat emptor.
I didn't know, so, just for anyone else who didn't:
> Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. It uses a human-readable, high-level build language. Bazel supports projects in multiple languages and builds outputs for multiple platforms. Bazel supports large codebases across multiple repositories, and large numbers of users.
There are many layers of caching within Bazel (remote/local, inmemory/disk), but the central functional incremental engine is called Skyframe [1]. Almost every computation within Bazel that can be incrementally executed is managed in this engine.
Provenance is a word I first saw advertised in a platform called dotscience.io — that I find fundamentally interesting. And it seems quite relevant to hermetic builds.
Provenance is about giving any state derived from an arbitrary computation an identity that is derived from the content hash of the inputs needed to re-compute that state ... in dotscience they achieve this by instrumenting io and creating zfs filesystem snapshots when computing new provenance artifacts.
I think this concept could be the ultimate building block for a build system — and it could become the job of oses/containers/runtimes/databases to Coordinate to allow this abstraction to be tracked with sufficient efficiency that programmers would feel allowed to freely use the concept of provenance when building ... it seems to me like provenance could provide all the information needed to support a distributed build cache? You wouldn’t actually need a build language at all — just an api in each language to ask for the saving of provenance artifacts. The artifact would hold all the info needed to be able to recompute the artifact with the same state — which is also all the info needed to decide when the artifact is out of date ...?
Build systems seem to sit in that category of perenial category of things that
keep getting re-invented, and either recapitulate existing problems or create
new ones.
I don't think people will ever fundamentally all agree on:
- static vs dynamic configuration
- custom language vs piggy back on existing
- intelligent, deeply integrated / understands code
it is building vs "language agnostic" but
necessarily shallow integration
All of these are fundamental tradeoffs that mean every tool will have
limitations that about 50% of people don't like. And so we
will keep re-inventing forever I think.
I hadn't heard of this, and see there is a lot of concern over using this project except for specific use cases.
I'm always weary of build tools that try to do multiple languages. On Scala projects I use SBT, and for people who have tried to hack on SBT itself or its plugins, you know it's a big mess under there. On other projects I've tried using Gradle with Scala, but I found a lot of times Gradle just wasn't setup for a Scala workflow or was missing essential tooling to make it as effective as SBT (although its configuration is considerably more sane). Most of the tooling and plugins around Scala are built around SBT as well (for better or for worse).
I try to stick with the major tool for a given language; cargo for Rust, SBT for Scala, the built-in tooling for go, with the exception of Java projects where I'd gladly take gradle over the hellscape that is Maven.
That works for small teams/projects. If you're looking at hundreds, thousand, or even tens of thousands person orgs you're spending time training everyone on every individual build system (very time-consuming and error-prone). Additionally because of the lack of consistency & unfamiliarity with having to deal with multiple interconnected projects, these tools fall apart spectacularly in that each team will end up with their own flavor of the build system. This makes transitioning between projects very hard & silos off teams.
That can be fine but can make it even more of an efficiency loss for someone switching projects/contributing partially to another project. Uniformity reduces costs on many fronts but like anything else it's a tradeoff. Now you need a team to maintain your Bazel/Buck/etc for each language & it may not jive 100% well with languages that have opinionated package managers/build systems alread (Node, Cargo, SBT, etc). On the other hand you'd probably end up having to create teams to maintain your company's Node, Cargo, SBT builds anyway except now you need to hire domain experts who not only understand each language but also how it should integrate within your larger infrastructure. A single uniform build system framework makes that easier.
I miss the days when JavaScript frameworks could be built with a simple npm install and executing a Grunt/Gulp file. Now to build Angular I need Yarn, Java, Bazel, and hundreds of megabytes of additional tooling downloaded by the build script. On a slow connection it takes ages to download everything, and even then the build often fails (on Windows I have yet to get it working successfully).
Edit: I'm referring to building the framework itself (e.g. to contribute a fix). Building an Angular project with the CLI works quite well.
malkia|6 years ago
With that, we end up paying so much extra time building everything over and over without need, and then not building things that we ought to.
So that's one reason to switch, but at the same time lots of people simply do not get it. To them it seems intrusive, new, opinionated, and makes them not happy to use it. I've used it for 2+ years at google, and yes initially - was WTF is this? Then it hit me... And I'm sure the same is for buck, pants, please.build, gn and other similar systems.
At the end of the day, you need way to express "end to end" your build graph, from any single individual source file, shell script, or configuration downto building your executables, deploying them, etc.
It's an industry tool, that needs to be looked, and if it takes 5 people to support it, then it takes 5 people to support it, but you won't be wasting other peeople's time on issues like - "Why this build in the CI did not trigger?", why it takes, and wastes my time (waiting for presubmit), etc.
Yes, it does not come for free, but it's worth knowing and trying it out at least.
If nothing else, here is the takeaway - Try to use a system with static graph, where relationships are known before you start building things. It's not always there, e.g. your #include "header.h" file is dynamic, but bazel forces you to express even that, and later it catches it whether you've done it, and breaks unless it's fixed.
klodolph|6 years ago
There’s an exercise you can do where you design a build system on the basis that it shouldn't do unnecessary work (which can be very slow and frustrating in practice).
My personal experience is that you can really quickly get to the point where just reading the entire graph into memory gets expensive. People talk about how Google is huge… but long before you get to that scale, you can end up with a build graph that just takes forever to parse and evaluate. (At Google's scale, it doesn't even fit in memory any more.)
So you decide that, as a hard design requirement, you should be able to only load the portion of the repository that you are building. And then you want to make this cacheable, so you can change the repository and know what’s changed in some quick / reasonable way.
If you go down this path, you end up rediscovering some of the big design decisions behind Bazel, Buck, Pants, Please, and GN.
habitue|6 years ago
- Do you have 200+ developers working on a monorepo?
- Are you willing to vendor all of your dependencies and maintain their builds yourself?
If so, consider it. The productivity you're losing to unnecessary rebuilding and re-running unchanged unit tests will probably be paid back if you can contort your development process to the one Bazel expects.
If you're a small shop, the benefits Bazel is going to provide over, say, Make (or whatever standard build system your primary language uses), are going to be minimal. And the overhead of maintaining Bazel is going to cost you a ton of developer time you may not be able to afford.
wereHamster|6 years ago
vlovich123|6 years ago
IshKebab|6 years ago
bobdobbs666|6 years ago
It is utter hell when you have tons of third party libraries (internal or external to the company) that you don’t have the source to and it is especially painful when trying to integrate bazels behavior against other build systems. Also bazels packaging and use of internal symlink renaming was a constant source of suffering. Bazel pretty much destroys a number of totally valid work linux commands for looking for so files.
Bazel might be useful in the case of a monorepo with a massive engineering pool AND a massive cloud infrastructure backing that repo to handle all the artifact sharing, but after having used cmake, premake, waf, random perl and ruby scripts, or just checking vs projects into perforce manually, I’d pick any of those before bazel for most projects. I say that having worked on code bases from a few 10s of thousands to 25+ million LoC with teams small, large, and distributed.
Bazel probably has its place but I have yet to find it.
klodolph|6 years ago
That said, the major sore point with Bazel for me is the general lack of expertise about how to work with it sanely. Depending on what part you’re looking at, it’s somehow both “too opinionated” and “too flexible” at the same time.
I think it will capture a big chunk of the mindshare for build systems over the next few years, and you’ll see more and more of it. Over that time, people will develop the expertise and best practices for different development problems.
For managing third-party dependencies specifically, Bazel gives you a ton of options, including options that only really make sense for huge orgs like Google. Google vendors their third-party libraries directly into the monorepo. If that doesn’t make sense for your org, Bazel lets you work with external Git repos, with artifact repositories, with package repositories like NPM, or with tools like pkg-config.
The thing that makes this hell, right now, is that few people how to use it well and the documentation is rough. I’m personally very happy with it, even for small codebases, but I’ve used it a lot.
ddevault|6 years ago
zellyn|6 years ago
Yes, it's complicated, but it's also quite rigorous, and the rigor pays off.
(We at Square had already found a Blaze-alike necessary. We are currently busy converting our Java build from Pants to Bazel.)
gravypod|6 years ago
Using Bazel with external packages on the other hand is one of the most tedious and frustrating endeavors imaginable. If you can vendor all of your source it's much less frustrating. This is extremely manageable in the C and C++ worlds where there aren't really any package mangers and you end up needing to do that anyway.
simfoo|6 years ago
Edit: I don't want to come off as too negative about Bazel. But I really think it needs more time and is nowhere near something that I would call a 1.0 let alone a 2.0
steeve|6 years ago
I’ve been using it for the last 2 years, and I am not doing any project again without it.
Bazel is not that complex if you start a project with it. Migrating to it and learning it at the same time will be hard though since you’re likely to uncover a lot of skeletons.
jacquesm|6 years ago
For very large organizations with the capability of assigning one or more teams to tooling it may very well be the right choice.
m0zg|6 years ago
ajconway|6 years ago
But which one? Are there any other (non blaze-like) build systems enabling hermetic (possibly remote) builds and caching?
If you don't need these properties and have a mono language project, the language's native build system sure fits and may be a better choice.
ZeroCool2u|6 years ago
I feel like I see the pattern of people on HN being disappointed in some of the tools that come out of Google and other large engineering orgs, when they don't work out really well in orgs that are not operating at the same scale. People have similar complaints about the complexity of other projects that come out of Google. K8s comes to mind as one such example. Often times these tools must be robust to such a large variety of uses that they are simply overkill for smaller organizations. I'll readily admit that I could be wrong and Bazel is simply poorly designed, but it is perhaps worth considering that the build system used by an engineering team of 50 need not be as complex as the build system used by one of the largest engineering orgs in the world. My guess is we'd see a lot less backlash if people tried to step off the hype train for a moment and critically evaluate whether they really need to use something like Bazel or K8s when something simpler would suffice.
dehrmann|6 years ago
s17n|6 years ago
orbifold|6 years ago
kylecordes|6 years ago
zegl|6 years ago
kryptiskt|6 years ago
lonelappde|6 years ago
This is a bizarre naming. They could call it Blaze installer 2.0 for Blaze 1.
brudgers|6 years ago
JadeNB|6 years ago
> Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. It uses a human-readable, high-level build language. Bazel supports projects in multiple languages and builds outputs for multiple platforms. Bazel supports large codebases across multiple repositories, and large numbers of users.
kovek|6 years ago
I was wondering how to make sure Bazel doesn’t rebuild something it has built previously? (Caching)
jingwen|6 years ago
[1]: https://bazel.build/designs/skyframe.html
breatheoften|6 years ago
Provenance is a word I first saw advertised in a platform called dotscience.io — that I find fundamentally interesting. And it seems quite relevant to hermetic builds.
Provenance is about giving any state derived from an arbitrary computation an identity that is derived from the content hash of the inputs needed to re-compute that state ... in dotscience they achieve this by instrumenting io and creating zfs filesystem snapshots when computing new provenance artifacts.
I think this concept could be the ultimate building block for a build system — and it could become the job of oses/containers/runtimes/databases to Coordinate to allow this abstraction to be tracked with sufficient efficiency that programmers would feel allowed to freely use the concept of provenance when building ... it seems to me like provenance could provide all the information needed to support a distributed build cache? You wouldn’t actually need a build language at all — just an api in each language to ask for the saving of provenance artifacts. The artifact would hold all the info needed to be able to recompute the artifact with the same state — which is also all the info needed to decide when the artifact is out of date ...?
dub|6 years ago
zmmmmm|6 years ago
I don't think people will ever fundamentally all agree on:
All of these are fundamental tradeoffs that mean every tool will have limitations that about 50% of people don't like. And so we will keep re-inventing forever I think.klysm|6 years ago
bru|6 years ago
djsumdog|6 years ago
I'm always weary of build tools that try to do multiple languages. On Scala projects I use SBT, and for people who have tried to hack on SBT itself or its plugins, you know it's a big mess under there. On other projects I've tried using Gradle with Scala, but I found a lot of times Gradle just wasn't setup for a Scala workflow or was missing essential tooling to make it as effective as SBT (although its configuration is considerably more sane). Most of the tooling and plugins around Scala are built around SBT as well (for better or for worse).
I try to stick with the major tool for a given language; cargo for Rust, SBT for Scala, the built-in tooling for go, with the exception of Java projects where I'd gladly take gradle over the hellscape that is Maven.
vlovich123|6 years ago
That can be fine but can make it even more of an efficiency loss for someone switching projects/contributing partially to another project. Uniformity reduces costs on many fronts but like anything else it's a tradeoff. Now you need a team to maintain your Bazel/Buck/etc for each language & it may not jive 100% well with languages that have opinionated package managers/build systems alread (Node, Cargo, SBT, etc). On the other hand you'd probably end up having to create teams to maintain your company's Node, Cargo, SBT builds anyway except now you need to hire domain experts who not only understand each language but also how it should integrate within your larger infrastructure. A single uniform build system framework makes that easier.
theodorejb|6 years ago
Edit: I'm referring to building the framework itself (e.g. to contribute a fix). Building an Angular project with the CLI works quite well.
teeray|6 years ago