History and motivation

For almost as long as there has been a Firefox browser, there has been a nightly update channel whereby bleeding edge developers could keep their browser up-to-date with the latest code changes from the past day. For lay people, imagine the security update notices you receive every month or so, but happening on a daily basis. There are changes made every day in the Mozilla codebase — bug fixes, new features, interface changes — so it’s important for us to have a vector to get these changes into testers’ hands as quickly as possible. Since the people that use these “nightly” builds are often developers or involved community members, the likelihood of us getting feedback on any code changes introduced in these builds is relatively high.

To minimize the download size of each nightly update for our testers, we offer what we call a “partial” update, where we provide only the accumulated binary code differences between the new nightly build and the previous one. If testers are more than one nightly update behind, we offer them a “complete” update, which is essentially just an archive of the latest nightly build packaged in a format that can be understood by the browser’s updating framework.

The developers and community members who use these nightly builds and updates don’t necessarily live in the U.S. or speak English as their first language, however. Armen took the first huge step towards better serving these users last year when he added nightly update support for localized builds (we call these builds l10n for short) on our core development branch, mozilla-central. For the first time ever, non-English users could get the same bleeding edge code as English speakers, but presented in their own language.

Useful, but at what cost

Mozilla has many different active code branches, all of which produce nightly builds for testing. Having proven its utility with the mozilla-central nightly builds, we wanted to extend the same l10n nightly update functionality to our other major branches, specifically the 1.9.2 and 1.9.1 code lines. However, after turning on nightly updates for the 1.9.2 branch, we soon discovered a bottleneck in our update process.

Since we first began offering nightly updates, all of our nightly updates had been generated on a single linux machine (or more recently, a single Linux VM). This was fine when there was only one locale to worry about, English (en-US). The update for each nightly build took between 2-3 minutes to generate, so even with 4 code branches (mozilla-central, 1.9.2, 1.9.1, CVS trunk) and 3 platforms (Linux, Mac OS X, Windows), updates would only ever take about 30 minutes to generate, total, and they would rarely be blocked because builds never finish at exactly the same time. Adding localized builds into the mix threw everything into disarray.

Mozilla prides itself on it’s localization story, but the sheer number of available localizations (around 75 per code branch) was swamping our poor, little update generation VM. As a rough estimate:

2.5 minutes/update x 3 update platforms x 75 locales
= 562.5 minutes
= >9 hours to generate all the nightly updates for one code branch

When we turned on the l10n nightly updates for the 1.9.2 branch, we had to make a number of changes immediately.

First, we certainly couldn’t consider turning on updates for another branch or nightly update generation would simply never catch up with itself. Because it would take over 18 hours to generate all the required updates for just two branches, we prioritized the English updates (en-US) so as not to degrade the existing nightly update experience for the consumers of English builds.

At this point, nightly updates for some locales were not timely at all, e.g. the Windows nightly build for the Chinese Traditional, Taiwan (zh-TW) locale would wait in the queue over 12 hours every day before having it’s partial update generated. In the timezones where a zh-TW localized build would most likely be used, this meant the base code of the “updated” build was always out-of-date by an entire day.

Adding a new platform, or even a new locale, was painful in a multiplicative rather than an additive way. Also, if we ever needed to re-spin a set of nightlies (which implies a new round of nightly updates), it meant that nightly update generation wouldn’t catch up for over 24 hours.

Make it right

Under these conditions, the decision to parallelize update generation was easy, the only question was how. We did consider simply adding more Linux VMs to generate updates, but that would have involved writing code to manage the pending update queue across multiple machines, or at the very least introducing an dedicated update generation machine per platform. The more straightforward solution would be to use the massive parallelization already provided by our existing build slave pool.

The first hurdle in generating our nightly updates on the build slaves was ensuring that the MAR (Mozilla ARchive) and binary diff tools worked on platforms other than Linux (which they now do) and were built automatically for branches on which we wanted to provide nightly updates (which they are now). There were also a bunch of build system changes that needed to happen to get MAR files uploaded automatically from the build slaves. After that, it was testing, lots and *lots* of testing. Last week, we finally got out of the testing phase and were able to deploy the changes for mozilla-central.

The numbers

There are always trade-offs when making a change like this, but honestly this is a pretty big win. Here are some typical l10n build scenarios from last week compared to the previous week before the updates-on-slave changes were in place:

In aggregate, we’ve added a maximum of 5 minutes/locale (often much less) x ~75 of locales, or 375 minutes, to the total time required to build all locales for a given branch on a given platform. Given 8 build slaves per platform assigned to create l10n nightly repacks, this means that the final l10n nightly repack on any build slave will finish roughly 45 minutes later than it did before the change. The parallelization of the overall process more than compensates though, at least as far as nightly updates are concerned. For example, the zh-TW localized Windows build may finish at 07:00 PDT now instead of 0615 PDT, but it’s nightly update is also immediately available at 07:00 PDT. English (en-US) builds also have their updates generated on the slaves, but since their updates were prioritized before anyway, they continue to be timely.

The localized builds also need to build the binary diff tool (bsdiff), but we re-use the same checkout for the tools from one l10n build to the next, so we only ever have to incur that build cost once, in the absence of a clobber. Even when we do need to rebuild bsdiff, since we also re-use the existing configuration steps and make targets that have already been run earlier in the l10n build process, the maximum time we ever spend building the bsdiff tool is about 1 min.

Onward

Perhaps what’s most exciting about this change are the possibilities it opens up for us, and for others.

Nick is already busy extending this work to provide nightly updates for project branches. Project branches are a relatively new thing in the Mozilla world, and soon the developers working on them will be able to stay current code-wise just the same as if they were working on mozilla-central. KaiRo also tells me that he’s only a few days away from having nightly partial updates available for SeaMonkey for the first time ever!

I hesitate to say that these consequences were unforeseen, but rather that we were initially unsure how/if the various systems we use to generate, store and serve updates would even cope when they had to deal with more than one locale.

We started by setting up a nightly update system in our release engineering staging environment, something that was long overdue anyway. Unfortunately, while this allowed us to test the functionality of our code changes, it didn’t allow us to test at anywhere near the scale we run at in our production environment. The staging update server was easily able to keep up with the trickle of new builds and mars coming in from a small pool of staging slaves.

Of course, when we moved from staging to production, having a single virtual machine (VM) producing updates for all locales across all branches quickly proved to be a formidable bottleneck. We started by only generating nightly updates for en-US and all locales on mozilla-central. We then started generating and serving updates for the new 1.9.2 branch shortly after it was created. At that point, we had to pause and take a good hard look at the time it was taking our single update VM to generate all the partial updates we were expecting of it.

Instead of only 9 partial updates per day (3 active code branches x 3 platforms, en-US only), we were now expecting the system to generate >400 partial updates (2 code branches with l10n nightly updates enabled x 3 platforms x ~70 locales) per day. Each update takes between 1 and 2 minutes to generate, so in a worst-case scenario, it could take up to 14 hours to create all the partial updates. If we had added another branch at that point, we might well have had gotten into a cycle where we never caught up with pending updates!

The nightly update generation script/cronjob had a few flaws that quickly became evident under the new load.

First, the script processed all pending updates in a single go when kicked off via cron. All the new builds that have been created since the last attempt are synced to the VM and then update generation commences. Any new builds that are created while the current iteration of the script is running have to wait until the entire current batch is finished and the crontab entry fires again.

Second, there was no prioritization of builds within the update generation script. Generally, it was first-in-first-out (FIFO) but it also depended on the vagaries of how the internal hash was created. This meant that en-US updates no longer occurred immediately, but whenever they happened to enter the queue. This turned out to be especially bad for Windows, our most popular platform.

Due to the increased cycle time due to profile-guided optimization (PGO), nightly builds on Windows often finish after 7am PDT. Unfortunately, all our l10n nightly repacks were scheduled to start at 7am PDT. This caused a few unfortunate problems. Sometimes l10n nightly repacks would repack the Windows build from the previous day, since the current nightly was still in progress. New l10n builds from other platforms would also often enter the queue for nightly update generation before the new en-US builds for Windows were ready. Windows nightly updates sometimes did not get processed until late in the afternoon! This was a serious regression for nightly testers.

A couple of quick fixes were available. Armen added some code to the nightly update generation script to sort the pending update graph so that any pending en-US updates in the current batch would always be processed first. I also changed the way l10n nightly builds were being created so that they are triggered as soon as the corresponding base en-US nightly is finished building. This allowed builds to enter the nightly update generation queue sooner (well, on Linux and Mac at least), but also fixed the longstanding problem on Windows of sometimes creating l10n repacks using the build from the previous day. Together, these changes sped up the entire nightly update generation process by about 2 hours in aggregate. They also allowed us to get nightly updates for en-US on Windows out before noon PDT. This is acceptable in the interim, but not ideal.

The timing is fortuitous though, since I had already planned to tackle other aspects of the nightly update generation system over the next few weeks. It is a natural extension of that work to change nightly update generation from a cronjob happening on a single VM to build step that happens on the build slave at the end of each nightly build. We already have this parallelism setup, so why not use it?

For any nightly testers out there, hopefully you can bear with us for a little while. We’ve greatly swelled your (potential) numbers by opening up nightly updates to the l10n community, but it will be a few more weeks until things are back to normal.

Last week, Armen flipped the configuration switch to turn on nightly updates for l10n builds on mozilla-central. After a few downtime-related hiccups last week that tanked nightly builds across the board, this morning we were finally able to serve partial updates to l10n nightly users on all platforms.

This is another huge step forward. Nightly testers who use localized builds no longer have to download an entirely new localized browser every day, but can instead rely on the Firefox update service to automatically download a much smaller binary partial update for them. Aside from allowing us to make sure l10n nightly testers are always testing the most recent code and string changes, this may also allow us to recruit a new batch of nightly l10n testers who were previously worried about the bandwidth involved in helping us test on an ongoing basis.

Over the coming weeks we’re hoping to enable l10n nightly updates on the newly-cut 1.92 branch as well. We are dealing with many more builds passing through the staging server to the update generation machinery now, so there’s some housekeeping to be done first to make sure that no builds, mars, or snippets are piling up in dark dusty corners anywhere.

Thanks to Armen for daring to touch the update code at all, to Nick for his update support, and to Axel for his tireless l10n work.

In fact, it was less than a year ago that Armen and John gave this presentation at the Summit in Whistler outlining the then-current problems with the l10n build system as it related to release engineering.

Axel has a new post up about how we’ve been systematically addressing those failings over the last 10 months, and what we’re tackling next.

We’re all about lowering bars to entry into the Mozilla community. I’m thrilled to see our localized Firefox builds approaching parity with the default English (en-US) builds. Not requiring potential localizers to have a complete translation before they can get meaningful feedback, and greatly reducing the turnaround time for that feedback to a few minutes is huge. Taking the stress of generating those builds off of Axel’s shoulders is also a big deal, because it gives him more time to work on other improvements for the l10n community.

There is more work still to do, certainly, but we’re a lot better off than a year ago. Cheers to Aki, Armen, Axel, John, and Seth for their help in getting us this far.