Working at Google, Blaze is one of the technologies that amazes me most. Any engineer can build any Google product from source on any machine just by invoking a Blaze command. I may not want to build GMail from source (could take a while) but it's awesome to know that I can.
I think this could be hugely useful to very large open source projects (like databases or operating systems) that may be intimidating for contributors to build and test.
Using Bazel (aka. Blaze) every day is one of the things that has made me dread ever leaving Google. Fast, reproducible builds are amazing. Once you have used this tool, it is very hard to go back. Personally, I'm thrilled that it has been open sourced.
I've been burned by so many build tools over the years. I've finally settled (for C/++/asm) on the combination of Make + ccache: I build a _very_ paranoid Makefile that recompiles everything if it feels like anything changes. For instance, every rule that compiles a C/++ file is invoked if _any_ header/inc/template file changes. I let ccache do the precise timestamp/check-sum based analysis. The result is that (for large builds < 10MMLOC) I rarely wait for more than a few hundred milliseconds on incremental, _and_ I have confidence that I never miscompile.
I just wish that I had a high-performance replacement for linking that was cross-platform (deterministic mode for ar), and for non-C/++ flows. Writing a deterministic ar is about 20 lines of C-code, but then I have to bake that into the tool in awkward ways. For generalized flows, I've looked at fabricate.py as a ccache replacement, but the overhead of spinning up the Python VM always nukes performance.
> I build a _very_ paranoid Makefile that recompiles everything if it feels like anything changes.
Do you have some kind of way to verify that your makefile dependencies conform to your source dependencies? Is clang/gcc tracking sufficient for your use case? What about upgrading the compiler itself, does your makefile depend on that? If so, how?
Have you considered tup[0]? Or djb-redo[1]? Both seem infinitely better than Make if you are paranoid. tup even claims to work on Windows, although I have no idea how they do that (or what the slowdown is like). Personally, I'm in the old Unix camp of many-small-executables, non of which goes over 1M statically linked (modern "small"), so it's rarely more than 3 secs to rebuild an executable from scratch.
> (deterministic mode for ar)
Why do you care about ar determinism? Shouldn't it be ld determinism you are worried about?
Convince your employer to ship a half decent unix environment with its OS and it will run on windows too. It's mostly a choice by microsoft to ship a half-baked command line interface with its products, you can't blame google for that.
Not really sure why this is getting downvoted - it's kind of an important detail when choosing a build system if you have to do multiplatform deployment.
s/not running Windows/not running Windows or refusing to install a free VM/
5 or 6 years ago I had to have Windows to run CAD software, but I found it easier to have a virtualbox install w/ Ubuntu in it for software development than trying to write code on Windows. The performance was good enough an the usability was pretty good. I imagine it has only gotten better since then.
Is Google departing from just throwing white papers over the wall and let community figure out the implementation details? blaze white paper was dropped a while ago and there are already two clones in Pants and Buck at Twitter and FB. It would be interesting to see how far off clones are from original implementation.
Getting rid of the timestamps in jar files is a huge improvement. I really hate it that when I recompile some huge java project I can't run a checksum on the jar to verify that the build is identical to a previous run (or when being dumped into some project that my current source tree is an accurate reflection of what is running in production).
Is Bazel developed fully in the open?
Unfortunately not. We have a significant amount of code
that is not open source; in terms of rules, only ~10% of
the rules are open source at this point. We did an
experiment where we marked all changes that crossed the
internal and external code bases over the course of a few
weeks, only to discover that a lot of our changes still
cross both code bases.
It's another impressive feat from Google and reading the comments I've kind of established that
1. Binaries are checked in to source
2. It's more structured than Gradle
3. It's for very large code bases
5. It's nix only
But...
1. We've already had the "chuck it in a lib directory" approach. The distributed approach maven/ivy etc seems to be working for the millions of developers out there who just have to get through the end of the day without production going up in flames. I suppose it's like moving a portion maven central into your code base. Checked in. Feels very odd, and kinda against one of the pillars of JVM: Maven. Love it or hate it it's one of most mature build/repository types out there. npm, bower anyone?
2. Got to agree with astral303. This isn't really something to shout about. Better reproducibility? Gradle/SBT have had incremental builds for quite a while. We all know there's no silver bullet, if you don't declare your inputs and outputs to gradle/blaze tasks or seed with random values then you're only going to get unrepoduceable builds.
3. Very large, I get that.
4. Very large code bases tend to enterprise systems. Enterprise systems tend to have a plethora of platforms/OSs so it being nix only is a drawback. However I suppose that if in charge of 10MLOC code base then I could mandate nix only builds? However in my experience they also tend to gravitate towards standards that seem to have longevity.
I'm yet to give it a go so I'll reserve final judgement. However I will say that I do wonder how far we'd be if Googles through their brightest minds at and worked with Maven/Gradle/SBT etc to scale their builds. (Yes I realise it's multi-lang - so is gradle). Perhaps the whole community would benefit from performance benefits.
Anyway hats off Google guys. It looks impressive and no doubt I'll jumping all over it in 12 months. In the mean time I'm off to go read up on Angular 2.0, or Typescript or ES6 or ES7 or whatever else I need* to know to get me through the day.
Really I'm just jealous I don't have 10MLOC code base :D
* If I have a Maven-based project with heavy reliance on pre-built jars from Maven Central, what's the recipe to port it to Bazel?
* Related, if I have multiple github repos, say a couple open source libraries and a couple private repos, what's a good recipe in conjunction to Bazel?
Check out http://bazel.io/docs/build-encyclopedia.html#maven_jar. In the root of your build, specify the jars you want from maven and then add them as dependencies in your BUILD files. The first time you run "bazel build", they'll be downloaded and cached from then on. It's somewhat limited in functionality at the moment, but should work for basic "download and depend on a jar".
Any reason the python support was ripped out? I've got my suspicions about not wanting/not being able to properly release the python packaging method in use internally, but I'm curious if I'd be tilting at windmills to try and get it to output pexes.
EDIT: I fully understand that this is a build tool for multiple languages. But its raison d'etre is speed. So I'm asking what techniques does Bazel use to accelerate builds and how do they differ from those used by sjavac, which is also designed to accelerate builds of huge projects?
> Why doesn't Google use …? Make, Ninja: These tools give very exact control over what commands get invoked to build files, but it's up to the user to write rules that are correct.
> Users interact with Bazel on a higher level. For example, it has built-in rules for "Java test", "C++ binary", and notions such as "target platform" and "host platform". The rules have been battle tested to be foolproof.
But does it give the optional custom level of control that for example CMake + Ninja provide? Or it's only high level rules?
[+] [-] habosa|11 years ago|reply
I think this could be hugely useful to very large open source projects (like databases or operating systems) that may be intimidating for contributors to build and test.
[+] [-] emu|11 years ago|reply
Using Bazel (aka. Blaze) every day is one of the things that has made me dread ever leaving Google. Fast, reproducible builds are amazing. Once you have used this tool, it is very hard to go back. Personally, I'm thrilled that it has been open sourced.
[+] [-] nevir|11 years ago|reply
[+] [-] malkia|11 years ago|reply
[+] [-] OneMoreGoogler|11 years ago|reply
A little too optimistic :) You can't build Android, Chrome, ChromeOS, iOS apps, etc. via blaze.
[+] [-] ambrop7|11 years ago|reply
[+] [-] solve|11 years ago|reply
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] thrownaway2424|11 years ago|reply
[+] [-] middleclick|11 years ago|reply
[+] [-] thechao|11 years ago|reply
I just wish that I had a high-performance replacement for linking that was cross-platform (deterministic mode for ar), and for non-C/++ flows. Writing a deterministic ar is about 20 lines of C-code, but then I have to bake that into the tool in awkward ways. For generalized flows, I've looked at fabricate.py as a ccache replacement, but the overhead of spinning up the Python VM always nukes performance.
[+] [-] beagle3|11 years ago|reply
Do you have some kind of way to verify that your makefile dependencies conform to your source dependencies? Is clang/gcc tracking sufficient for your use case? What about upgrading the compiler itself, does your makefile depend on that? If so, how?
Have you considered tup[0]? Or djb-redo[1]? Both seem infinitely better than Make if you are paranoid. tup even claims to work on Windows, although I have no idea how they do that (or what the slowdown is like). Personally, I'm in the old Unix camp of many-small-executables, non of which goes over 1M statically linked (modern "small"), so it's rarely more than 3 secs to rebuild an executable from scratch.
> (deterministic mode for ar)
Why do you care about ar determinism? Shouldn't it be ld determinism you are worried about?
[0] http://gittup.org/tup/
[1] https://github.com/apenwarr/redo
[+] [-] justicezyx|11 years ago|reply
[deleted]
[+] [-] latkin|11 years ago|reply
[+] [-] jacquesm|11 years ago|reply
[+] [-] izacus|11 years ago|reply
[+] [-] krschultz|11 years ago|reply
5 or 6 years ago I had to have Windows to run CAD software, but I found it easier to have a virtualbox install w/ Ubuntu in it for software development than trying to write code on Windows. The performance was good enough an the usability was pretty good. I imagine it has only gotten better since then.
[+] [-] sangnoir|11 years ago|reply
[+] [-] EricBurnett|11 years ago|reply
If you've got the expertise to add it, sounds like it would be welcomed.
[+] [-] mellery451|11 years ago|reply
[+] [-] cbhl|11 years ago|reply
[+] [-] spankalee|11 years ago|reply
[+] [-] tkubacki|11 years ago|reply
[+] [-] ngd|11 years ago|reply
I know it as Blaze, which Bazel is an anagram of. Many files in the source have references to Blaze.
[+] [-] mashraf|11 years ago|reply
[+] [-] cbgb|11 years ago|reply
[+] [-] jacquesm|11 years ago|reply
[+] [-] zobzu|11 years ago|reply
It seems like a stricter, huge make-like harness (in fact it reminds me of the mozilla firefox python build system a bit).
It's not bad by any means, but it seems like to me it doesn't "magically" fix the "be reproducible" problem at all (which is what it seem to claim)
Am I missing something?
[+] [-] yarapavan|11 years ago|reply
[+] [-] cies|11 years ago|reply
I read in the "Getting started":
> You can now create your own targets and compose them.
So does this mean it is a replacement for `make`? => Yes
Found the answer here: http://bazel.io/docs/FAQ.html
[+] [-] kchod|11 years ago|reply
[+] [-] w4tson|11 years ago|reply
1. Binaries are checked in to source 2. It's more structured than Gradle 3. It's for very large code bases 5. It's nix only
But...
1. We've already had the "chuck it in a lib directory" approach. The distributed approach maven/ivy etc seems to be working for the millions of developers out there who just have to get through the end of the day without production going up in flames. I suppose it's like moving a portion maven central into your code base. Checked in. Feels very odd, and kinda against one of the pillars of JVM: Maven. Love it or hate it it's one of most mature build/repository types out there. npm, bower anyone?
2. Got to agree with astral303. This isn't really something to shout about. Better reproducibility? Gradle/SBT have had incremental builds for quite a while. We all know there's no silver bullet, if you don't declare your inputs and outputs to gradle/blaze tasks or seed with random values then you're only going to get unrepoduceable builds.
3. Very large, I get that.
4. Very large code bases tend to enterprise systems. Enterprise systems tend to have a plethora of platforms/OSs so it being nix only is a drawback. However I suppose that if in charge of 10MLOC code base then I could mandate nix only builds? However in my experience they also tend to gravitate towards standards that seem to have longevity.
I'm yet to give it a go so I'll reserve final judgement. However I will say that I do wonder how far we'd be if Googles through their brightest minds at and worked with Maven/Gradle/SBT etc to scale their builds. (Yes I realise it's multi-lang - so is gradle). Perhaps the whole community would benefit from performance benefits.
Anyway hats off Google guys. It looks impressive and no doubt I'll jumping all over it in 12 months. In the mean time I'm off to go read up on Angular 2.0, or Typescript or ES6 or ES7 or whatever else I need* to know to get me through the day.
Really I'm just jealous I don't have 10MLOC code base :D
[+] [-] setheron|11 years ago|reply
[+] [-] malkia|11 years ago|reply
[+] [-] asuffield|11 years ago|reply
[+] [-] pacala|11 years ago|reply
* If I have a Maven-based project with heavy reliance on pre-built jars from Maven Central, what's the recipe to port it to Bazel?
* Related, if I have multiple github repos, say a couple open source libraries and a couple private repos, what's a good recipe in conjunction to Bazel?
[+] [-] kchod|11 years ago|reply
For multiple Github repos, use http://bazel.io/docs/build-encyclopedia.html#http_archive or http://bazel.io/docs/build-encyclopedia.html#new_http_archiv... (depending on if it's a Bazel repository or not). Let us know if you have any questions or issues!
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] mikojava|11 years ago|reply
https://www.gradle.org/gradle-team-perspective-on-bazel/
[+] [-] pjjw|11 years ago|reply
[+] [-] cromwellian|11 years ago|reply
[+] [-] solomatov|11 years ago|reply
[+] [-] pron|11 years ago|reply
EDIT: I fully understand that this is a build tool for multiple languages. But its raison d'etre is speed. So I'm asking what techniques does Bazel use to accelerate builds and how do they differ from those used by sjavac, which is also designed to accelerate builds of huge projects?
[+] [-] Zariel|11 years ago|reply
[+] [-] shmerl|11 years ago|reply
> Users interact with Bazel on a higher level. For example, it has built-in rules for "Java test", "C++ binary", and notions such as "target platform" and "host platform". The rules have been battle tested to be foolproof.
But does it give the optional custom level of control that for example CMake + Ninja provide? Or it's only high level rules?
[+] [-] blinks|11 years ago|reply
You can [at least internally] define custom rules to handle pretty much anything, in almost-but-not-quite-python.
[+] [-] frownie|11 years ago|reply
Multi-language support: Bazel supports Java, Objective-C and C++ out of the box, and can be extended to support arbitrary programming languages.
c'mon, not even the Go language from Google itself ?