On October 23, the day before the first European BazelCon, EngFlow and Tweag organized the sixth Bazel Community Day at the Salesforce office in Munich, capped off with a happy hour sponsored by Gradle.

The event kicked off early afternoon with snacks and welcome talks:

• Introduction by Helen Altshuler (EngFlow) and Andreas Herrmann (Tweag)
• Bazel at Salesforce by Gunnar Wagenknecht (Salesforce)

After that, the event split into two parallel tracks:

Track 1:

• A Bazel Beginner Bootcamp, by Billy Autrey (EngFlow)
• Debugging Cache Misses and Sources of Nondeterminism in Bazel, by Ben Radford (Tweag) and Joseph Gette (Mercedes)

The Bootcamp was delivered to a packed room with mixed experience levels and received useful feedback which will be used to improve the next one.

It was really good actually. I already use Bazel, but the Bootcamp helped me fill in some gaps.

The session on debugging cache misses was also in high demand. We had to move it to the main stage to provide enough room for all attendees. This session also received positive and constructive feedback.

Track 2:

Comprised a series of lightning talks:

• Coverage with Bazel: An Overdue Summary, by Ulf Adams (EngFlow)
• Bazel Migration Using Fully Ephemeral BUILD Files, by Markus Hofbauer (Luminar)
• Fast Incremental Bazel Builds with ‘Persistent Pods’, by Shishir Kumar (ThoughtSpot)
• Build Server Protocol, by Andrzej Głuszak (Jetbrains)
• Buck2: Optimizations and Dynamic Dependencies, by Neil Mitchell and Chris Hopman (Meta)
• Bazel + Go with rules_go and Gazelle, by Tyler French and Zhongpeng Lin (Uber)
• Building JavaScript & TypeScript with rules_js, by Greg Magolan (Aspect)

There were also two unconference sessions. One focused on managing large monorepos and remote execution, while the other went into a deeper discussion on Buck2, Bazel, Starlark, and bzlmod. Bazel folks were interested to know more about Buck2’s Starlark type checking, profiling, debugging, and dynamic actions.

We’ll go into more detail about some of those talks and sessions. The recordings of the lightning talks are available on YouTube.

Intro and welcome

Helen Altshuler (EngFlow) gave us a brief history of Bazel Community Day, a quick recap of the Build Meetup from a few days prior, and some numbers about Bazel uptake. She also mentioned that the next Community Day would be held in London, sometime in June 2024.

Andreas Herrmann (Tweag) gave us a breakdown of the Community Day Structure and logistics for the day and moderated the lightning talks.

Gunnar Wagenknecht (Salesforce) gave us a detailed account of the scale of the Bazel migration at Salesforce. Many groups are using Bazel, some with up to 4000 engineers, multiple languages and operating systems, and up to 120000 Bazel targets. This was made possible by disallowing custom build rules, regulating load statements, and testing everything.

Coverage with Bazel: An Overdue Summary, by Ulf Adams (EngFlow)

After excitedly confessing to having had two colas before the talk, Ulf Adams sped into an overview of code coverage and Bazel.

Coverage keeps track of what code is executed.

Ulf underscored that while low coverage may indicate subpar tests, a high coverage figure doesn’t guarantee the quality of the tests. You also need to select tests based on coverage to target your changes to optimize resource utilization.

He went on to talk about how there is no right way to collect coverage, instead, it can be done:

• per file
• per class
• per function
• per line
• per character

When looking at code coverage with Bazel, you find that different languages have different coverage stacks, each with its own format! Before having a human look at the output you’re probably going to need to post-process it:

1. With an output generator,
2. A report generator, or
3. By manually generating the HTML for human consumption.

Bazel Migration Using Fully Ephemeral BUILD Files, by Markus Hofbauer (Luminar)

Markus gave us an account of how Luminar, a lidar system built from chip for autonomous driving, migrated to ephemeral BUILD files that aren’t checked into version control.

They were initially using Conan, a C++ package manager, to instrument CMake and split the monolithic CMake project into packages with interdependencies. That resulted in a handcrafted build system for the mono repo, which didn’t work for them in the end.

Then they looked at automatically converting the Conan Buildfile into Bazel BUILD files, but:

• people introduced changes that broke the build, and
• they didn’t want to maintain two CI systems.

So how did they solve this? By always auto-generating the Bazel BUILD files, using Gazelle, with the following constraints:

• always mapping 1 header file to 1 target, resulting in small targets
• figuring out dependencies by looking at include files with naming conventions
• handling edge cases with manual targets that override Gazelle
• using NixOS for 3rd party dependencies
• removing local BUILD files

All of which led them to a happy state with low Bazel maintenance so they could focus on developing new features.

Fast Incremental Bazel Builds with ‘Persistent Pods’, by Shishir Kumar (ThoughtSpot)

Shishir’s opening line,

What do you want in life? Peace, long life, money, and faster builds.

got some laughs from the crowd. He went on to discuss the difference between clean and incremental builds, and how he leveraged warm Bazel caches with long-living Kubernetes pods as Jenkins agents, and local disk caches as low-latency persistent volumes to achieve a 40% reduction in average Build and Test time.

Challenges included finding the right pod for the incoming build, minimizing the amount of work that you do, and syncing Git branches.

Ongoing work includes experimenting with different heuristics to minimize build time:

• using the SHA from the previous build
• diffing the targets between the two SHAs and using a smaller target list

All in all, they made good progress, but there are many avenues still left to explore.

Build Server Protocol, by Andrzej Głuszak (Jetbrains)

Andrzeij presented a lightning view of the Build Server Protocol (BSP), which creates an abstraction layer between editors and build tools, analogous to what the Language Server Protocol (LSP) does for language tools.

By using the protocol, you can write once and run everywhere! Andrezej also mentioned a new IntelliJ plugin for Bazel that uses BSP.

Buck2: optimizations and dynamic dependencies, by Neil Mitchell and Chris Hopman (Meta)

Next up, Neil and Chris gave an account of Buck2, the new primary build system at Meta, which is:

• Polyglot
• For Monorepos
• Open Source
• Twice as fast as Buck1

Buck2 has target files in Starlark specific to the project, rules which are all written in Starlark, a core binary written in Rust, providing the Starlark APIs and a Starlark interpreter. It also has profiling, linting and a type checker built in.

The main reason to build Buck2 was to go faster, and as such, it has:

• a single dependency graph with no phases
• remote execution, with pre-computed Merkle trees, and virtual files to provide builds without the bytes and inputs through EdenFS.

Some rules don’t yet work, and it does not have a bzlmod-like package manager, but it does have dynamic dependencies and the Buck Extension Language (BXL).

You can learn more about the differences and similarities between Bazel and Buck2 here.

Bazel + Go with rules_go and Gazelle, by Tyler French and Zhongpeng Lin (Uber)

Tyler and Lin talked about how nogo, the rules_go static analysis tool, ran as part of the compile action, and how changing the nogo settings file invalidated the cache. An approach to solving this could be moving the nogo check out of the compile action.

They also talked about using Go modules with generated code, and handling dependencies with large MODULE.bazel files.

Building JavaScript & TypeScript with rules_js, by Greg Magolan (Aspect)

Greg started by describing the early state of JavaScript support in Bazel, which came out in 2017 with rules_nodejs but hit a dead end in terms of performance with third-party dependencies and runtime compatibility with JavaScript and TypeScript build tools.

In 2021, Greg and Alex Eagle started writing rules_js, which was released in August 2022 and has gained steady adoption.

rules_js handles third-party dependencies much more efficiently, by working with pnpm lockfiles.

Debugging Cache Misses, by Ben Radford (Tweag), and Joseph Gette (Mercedes)

Ben and Joseph co-presented a workshop on debugging cache misses.

It began with a brief explanation of hermeticity and determinism, which are important concepts to know about when trying to understand why a cache miss has occurred.

Then the audience were invited to clone the [toy repository][cache-misses-worksho] created for the workshop, and follow along as various issues were discovered, explained, and fixed.

Bonus: Games Night after BazelCon Day 1

The next day, in the evening after BazelCon Day 1, JetBrains hosted a games night at their Munich office, co-organized with EngFlow.

Spirits were high after a packed and exciting first conference day. Groups formed where people chatted with beverages in hand or played board games, card games, and chess.

We even had a chance to enjoy a round of a Jeopardy-style quiz game about Bazel flags, featuring the new deck of Bazel flag playing cards EngFlow printed for the event.

Bringing Bazel Community Day to You

If you’d like to join a future Bazel Community Day, then you can fill in this survey hosted by EngFlow.

You can also read this article on the EngFlow Blog.