040coders.nl: Talks and Videos. Food and Drinks. Every 3rd Thursday somewhere in the 040 region. 040coders.nl: Talks and Videos. Food and Drinks. Every 3rd Thursday somewhere in the 040 region.

Distributed software builds

using the REv2 protocol


Ed Schouten By: Ed Schouten

Talk at Meetup 20200618


Once a software project becomes large enough, you often see that the developer experience becomes painful. Builds tend to take ages. In the worst case, it can even become impossible to do a full 'build and test' on your personal machine. A full build may depend on different hardware platforms and operating systems. You often see that people then resort to building just parts of the project on their own system, and hope that CI (Continuous Integration) catches any remaining problems. This is of course undesirable.

Blaze is a build system that was developed inside of Google to solve this problem. What makes Blaze an interesting tool is that it only orchestrates builds. Instead of running build actions (e.g., compilers) on your own system, it sends requests to a build cluster. Builds become a lot faster as a result of that, both because clusters are faster than a single workstation and because they can maintain a centralized cache.

In 2015, Google released an Open Source version of Blaze called Bazel. One of the highlights of the project has been a community-driven effort to design an open build cluster protocol, named 'Remote Execution v2' (REv2 for short). It looks like this protocol is becoming the de facto standard for doing remote builds. There are now various clients (Bazel, Pants, Goma, Recc, Buildstream) and servers (Buildbarn, Buildfarm, Buildgrid and Google Cloud RBE) that implement it.

In this talk I'm going to discuss how REv2 works. By the end of this talk you will have a good understanding of the data model that Bazel and Buildbarn use, and how they exchange inputs (source code) and outputs (object files, executables).

Ed's slides are now available


Ed Schouten is an Open Source developer from the Netherlands. He has been a contributor to the FreeBSD project since 2008. One of his major contributions was the import of the Clang C++ compiler into the base system. In 2018, Ed started working on the Buildbarn project; an Open Source remote build cluster.

contact: organizers at 040coders.nl
contact: organizers at 040coders.nl