The new meeting format is working pretty well for us so far. Here are some of the key things I like about it:

• The framework makes it easy to decide when in the meeting to discuss which issues. The project-based agenda I used previously tended to encourage digression, but that’s kept to a minimum now. Certain meeting sections are pure reporting (e.g. accomplishments) and others are meant for discussion (e.g. blockers).
• It puts an onus on both parties to be prepared. Armen sends his notes to me the night before so I always have a chance to prepare help/advice beforehand.
• The video chat is great as a remotie. Armen has an office he can go in to in Toronto, but I have no such luxury. Having these meetings “face-to-face” (i.e. Skype) is a big win, at least for me. Non-verbal communication can help us move faster through the agenda at times. Conversely, it can also allow me to pick up on signs that something may be amiss. Both functions are invaluable.

Our success with the format has encouraged me to push the format back up the chain: John and I are going to try the format next week for our 1-on-1.

I whipped up a dead-simple greasemonkey script today that increases the size of the comment box when you click on it, and then shrinks it back down again when you click elsewhere.

• Download: GrowReviewComment

In response to the comments on my previous entry about Tackling the Release Engineering:Future queue and a number of conversations I’ve had out-of-band over the past few months, the release engineering team has decided to do away with the Future component. We will be merging all of the bugs therein back into the regular Release Engineering component.

Aside from the simplicity inherent in having fewer components to worry about, there were three main reasons we chose to do this:

1. No other group in the company/community has an explicit “Future” component where they can hibernate bugs.
2. People outside the RelEng team find the Future component opaque, and despite reassurances to the contrary, they often just assume bugs in the Future component won’t be fixed.
3. Moving bugs between the regular Release Engineering component and the Future component makes it very hard to extract meaningful statistics about how our bugs change over time (new bug rate, fix rate, etc.). This makes it hard to change our processes based on data.

What does this mean for you? Nothing, hopefully. I’m planning to move all the bugs out of the Future component on the morning (EST) of Monday, February 15th. People in North America will be on holiday, so this should be a relatively low-traffic time to make this change. My apologies in advance for the RelEng mail bomb you’ll be getting on Tuesday morning. :/

We will still continue to triage bugs using our three bucket approach, only the location of the third bucket will have changed. Our three buckets of bugs correspond to the following:

1. Untriaged: – state: NEW, assigned to: nobody, priority: none (–)
2. Triaged, assigned – state: NEW (we’re not working on it yet) or ASSIGNED (we’re working on it), assigned to: a specific release engineer, priority: P1-3
3. Triaged, unassigned (was Future) – state: NEW, assigned to: nobody, priority: P3-5

In the interest of further simplification, the RelEng team will eschew the Severity field as we move forward, although we won’t change any information already contained in that field if it exists. Instead, we’ll start using the Priority field more diligently. All bugs that have been triaged will have a Priority associated with them from here on out.

For reference, the various Priority levels mean the following to us:

P1: blocker, all other work on hold until this is fixed
P2: being actively worked on
P3: acknowledged as an issue, but not being actively worked on
P4: (future) goals that are either blocked on other work, or that simply have not been started yet
P5: enhancement

There was some concern within the RelEng team that developers might assume that bugs moved out of the Future component were now being worked on. I have faith that people will correctly interpret the implications of their bug still being assigned to nobody@mozilla.org as an indication that, no, their bug is still on the shelf.

Please note that this does not change the triage work that still needs to happen on these once-and-Future bugs, as described in the previous post. We’ve already made some good headway on triage this week, though. The number of “Future” bugs is now down to 304 from a high of 343. Kudos to the people who made that happen. Let’s keep up that great work.

I’m certain most developers first response to having their bug moved to the Future queue is, “Oh, look, my bug has fallen down a well.” Historically speaking, that may not be far from the truth.

What goes in Future?

Why does the Future component make people get punchy? For a long time, the Future component has lacked a decent description, so developers don’t know what it means when their bugs are moved there. Many have started hording their gigawatts in anticipation.

Bugzilla currently has the following small description of the Release Engineering:Future component:

“For longer term projects that have been agreed should be done, but have no immediate plans to so. These are not be part of the regular recurring triage. Advanced planning and placeholder goals for next quarter also go here.”

Despite the grammatical errors, this description is mostly accurate, but what does it actually mean:

• Triaged bugs with no immediate owners go here.
• Sadly, most enhancement bugs end up here unless they will make the core release process better, more streamlined, etc. quickly.
• Bugs that are blocked on longer term projects in other groups go here until there is something for RelEng to do.
• Advanced planning and placeholder goals for next quarter (or later) also go here. That’s pretty self-explanatory.

Remember: this description is for the Future component ONLY. The RelEng team continues to pick up and work on bugs that need to get done on a daily basis.

I’m thinking about how to improve this description, and will get the description updated in Bugzilla once I have achieved some rough consensus. In the meantime, I’ve posted this description of the Future component in the wiki as an ongoing reference.

The (Ever-Increasing) Numbers

At the time of writing, the Release Engineering:Future component has 343 bugs in it. This number has grown steadily over the past year despite having more release engineers on staff, and having made a great many improvements to our release automation and our continuous integration infrastructure.

Our turnaround time for bugs in the Future component is also not stellar. At our urging, the Mozilla Metrics team recently started setting up a dashboard to give us various statistics on our Bugzilla usage. Bugs don’t get fixed in the Future queue, so it’s hard to make truly accurate assessments here, but there are a lot of aging bugs in there. According to the numbers, fully 50% of the bugs in the Future component are older than 1 month and more than 25% are older than 6 months. How this compares to other teams or areas, I can’t say, but it certainly makes me empathize with developers who feel their Future-ed bugs have gone walkabout.

If it makes you feel better in a sadistic way, the majority of bugs filed by RelEng team members go directly into the Future queue. We’re not overly happy about it either.

I have a way? Is that better than a plan?

Things are getting done, though. For example, over the past month, the number of non-Future, release engineering bugs that are actively being worked on has gone from a low water mark of 130 up to over 200 today. For comparison, over the same period, the total number of bugs in the Future queue has slowly crept up from a low of 302 to 323. Mozilla is growing along so many axes that sometimes it feels like keeping the increase in the number of Future bugs to a linear relationship rather than exponential one is an accomplishment in itself.

So how do we stop the increase and start wrestling the Future component back under control?

The first step is triage. Starting this Thursday, February 11th, the RelEng team is going to have bi-weekly triage meetings to specifically prune down the Future queue.

As part of the triage, we’re going to be touching every bug and updating the whiteboard field with searchable tags. Our goal here is to make it easy to find classes of bugs in the Future queue so that duplicates and overlap can be easily eliminated, and fixes can be batched as much as possible.

We’ll be keeping a list of the tags we’re using in the wiki in case you want to follow along.

There are some classes of bugs that we won’t be able to eliminate (e.g. future goals), but hopefully within a few months, we’ll have the Future queue back under control.

It won’t be a quick fix, but it’s one we’re committed to.

Since we implemented the RelEng sheriff (or “buildduty” as it is more informally called) last spring, developers have been getting better about using the buildduty person as the first point of contact for RelEng issues. I’d like to implore people to continue doing so. There are a few exceptions, of course:

• It is outside of regular work hours and/or the designated buildduty person is not available; or,
• you are already working on a bug with a specific release engineer.

We recognize that different release engineers have different core competencies, and it may be tempting to go “straight to the source,” but please respect our process. The process is in place to limit (as much as possible) the interrupt-driven nature of release engineering work and to minimize the impact of those interrupts on ongoing projects. In many cases, the person with the expertise you need will still end up helping you (we’re all in the same IRC channels, after all), but the buildduty person is specifically committed to helping you, even if only as a thin scheduling proxy for another release engineer.

Ben’s post had a good list things that RelEng can help developers with. Based on actual requests over the past few months, here are a few other services we can offer:

• clone a virtual machine for one-off debugging/testing: Sometimes patches work on the try server but (for whatever reason) don’t once checked in. Sometimes tests fail or are intermittently orange. If you’re seeing issues like these after landing a patch, or need access to a specific platform for short-term work, the RelEng team can clone one of our reference machines and give you access (under certain circumstances).
• grab build/extra logs from a build machine (provided they exist). This is often useful if a try server build generates extra, non-standard output

In both the above cases, the best thing to do is talk to the person on buildduty and then file a bug.

Conspicuously missing from Ben’s post was a list of what RelEng can’t do for you. While we can usually point you in the right direction, there are some things that we explicitly cannot do:

• help you debug issues with your own local builds. We’re not (necessarily) build config experts, so unless you’re running with exactly the same configuration as our build machines and don’t have any code patches applied, you’d be better served asking the talented folk in #developers or posting to the newsgroups (m.d.builds or m.d.a.firefox)
• shepherd your patch through the landing process. Standard patch landing rules apply: *you* are expected to be around when you land patches. If builds/tests fail and you’re not around, your patch will likely be backed out.

Need help finding the RelEng sheriff? Here’s the RelEng sheriff schedule.

Our focus will remain the same: look for large-ish sections of RelEng-related project work, whether existing or new, and drive them to completion. Some things we’ve worked on previously include properly integrating l10n builds into our buildbot infrastructure, and enabling nightly updates for localized builds. We are currently working on standardizing on one set of update tools and getting localized builds put together for mobile.

John has affectionately dubbed our sub-group “Big Chunks” in reference to the scope of work we take on. That’s a terrible name, and that’s where you come in. I’m soliciting alternate naming ideas for our new group. The leading candidate so far was proposed by Armen: “Chunk Norris”. I have faith in the collective wisdom of the internet to come up with something better (perhaps not even referencing “chunks” at all) or at the very least to entertain me with your attempts.

And….GO!

I’ve done my share of releases over the past few quarters, and until the current release (3.5.4, still in progress as of writing), I would have said that I understood the process pretty well.

The RelEng team has a convention when documenting our release procedures. When a given step completes successfully, we mark it with ‘No problems’ in the build notes. The problem with ‘No problems’ is that it isn’t instructive.

The current 3.5.4 release, for which I’m responsible on the RelEng side, has been about as far from ‘No problems’ as I can reasonably imagine. The in-progress build notes are available here for the rubberneckers and/or morbidly curious. We’ve had respins, we’ve had aborted betas, we’ve had automation interrupted at the very first step by tagging collisions, we’ve had single-locale rebuilds. I’ve cursed each and every one of these things as they’ve happened, but I’ve muddled through them with the help of others on my team. In the process, I’ve learned far more about the underlying processes than I ever would have if our release automation had simply chugged along to completion.

It turns out what I understood about our release procedure was how to follow a recipe. As long as I didn’t run out of ingredients and had all the right pots and pans, the recipe came out just as expected. Fortunately (and yes, I am going to say fortunately here because that’s the whole point), this current release has enabled me to ad-lib in the release kitchen a lot more when things are not perfect. Now if only I could find the release automation equivalent of bacon so I can declare culinary victory much earlier in the release process…but perhaps that’s where this metaphor breaks down.

Don’t get me wrong, every time I do a release, I’ll still be praying to anyone that will listen for build steps to complete with ‘No problems,’ but I no longer fear the occasional and inevitable problems that will crop up in a system of this complexity.

I hesitate to say that these consequences were unforeseen, but rather that we were initially unsure how/if the various systems we use to generate, store and serve updates would even cope when they had to deal with more than one locale.

We started by setting up a nightly update system in our release engineering staging environment, something that was long overdue anyway. Unfortunately, while this allowed us to test the functionality of our code changes, it didn’t allow us to test at anywhere near the scale we run at in our production environment. The staging update server was easily able to keep up with the trickle of new builds and mars coming in from a small pool of staging slaves.

Of course, when we moved from staging to production, having a single virtual machine (VM) producing updates for all locales across all branches quickly proved to be a formidable bottleneck. We started by only generating nightly updates for en-US and all locales on mozilla-central. We then started generating and serving updates for the new 1.9.2 branch shortly after it was created. At that point, we had to pause and take a good hard look at the time it was taking our single update VM to generate all the partial updates we were expecting of it.

Instead of only 9 partial updates per day (3 active code branches x 3 platforms, en-US only), we were now expecting the system to generate >400 partial updates (2 code branches with l10n nightly updates enabled x 3 platforms x ~70 locales) per day. Each update takes between 1 and 2 minutes to generate, so in a worst-case scenario, it could take up to 14 hours to create all the partial updates. If we had added another branch at that point, we might well have had gotten into a cycle where we never caught up with pending updates!

The nightly update generation script/cronjob had a few flaws that quickly became evident under the new load.

First, the script processed all pending updates in a single go when kicked off via cron. All the new builds that have been created since the last attempt are synced to the VM and then update generation commences. Any new builds that are created while the current iteration of the script is running have to wait until the entire current batch is finished and the crontab entry fires again.

Second, there was no prioritization of builds within the update generation script. Generally, it was first-in-first-out (FIFO) but it also depended on the vagaries of how the internal hash was created. This meant that en-US updates no longer occurred immediately, but whenever they happened to enter the queue. This turned out to be especially bad for Windows, our most popular platform.

Due to the increased cycle time due to profile-guided optimization (PGO), nightly builds on Windows often finish after 7am PDT. Unfortunately, all our l10n nightly repacks were scheduled to start at 7am PDT. This caused a few unfortunate problems. Sometimes l10n nightly repacks would repack the Windows build from the previous day, since the current nightly was still in progress. New l10n builds from other platforms would also often enter the queue for nightly update generation before the new en-US builds for Windows were ready. Windows nightly updates sometimes did not get processed until late in the afternoon! This was a serious regression for nightly testers.

A couple of quick fixes were available. Armen added some code to the nightly update generation script to sort the pending update graph so that any pending en-US updates in the current batch would always be processed first. I also changed the way l10n nightly builds were being created so that they are triggered as soon as the corresponding base en-US nightly is finished building. This allowed builds to enter the nightly update generation queue sooner (well, on Linux and Mac at least), but also fixed the longstanding problem on Windows of sometimes creating l10n repacks using the build from the previous day. Together, these changes sped up the entire nightly update generation process by about 2 hours in aggregate. They also allowed us to get nightly updates for en-US on Windows out before noon PDT. This is acceptable in the interim, but not ideal.

The timing is fortuitous though, since I had already planned to tackle other aspects of the nightly update generation system over the next few weeks. It is a natural extension of that work to change nightly update generation from a cronjob happening on a single VM to build step that happens on the build slave at the end of each nightly build. We already have this parallelism setup, so why not use it?

For any nightly testers out there, hopefully you can bear with us for a little while. We’ve greatly swelled your (potential) numbers by opening up nightly updates to the l10n community, but it will be a few more weeks until things are back to normal.

Last week, Armen flipped the configuration switch to turn on nightly updates for l10n builds on mozilla-central. After a few downtime-related hiccups last week that tanked nightly builds across the board, this morning we were finally able to serve partial updates to l10n nightly users on all platforms.

This is another huge step forward. Nightly testers who use localized builds no longer have to download an entirely new localized browser every day, but can instead rely on the Firefox update service to automatically download a much smaller binary partial update for them. Aside from allowing us to make sure l10n nightly testers are always testing the most recent code and string changes, this may also allow us to recruit a new batch of nightly l10n testers who were previously worried about the bandwidth involved in helping us test on an ongoing basis.

Over the coming weeks we’re hoping to enable l10n nightly updates on the newly-cut 1.92 branch as well. We are dealing with many more builds passing through the staging server to the update generation machinery now, so there’s some housekeeping to be done first to make sure that no builds, mars, or snippets are piling up in dark dusty corners anywhere.

Thanks to Armen for daring to touch the update code at all, to Nick for his update support, and to Axel for his tireless l10n work.

Because Ted is more about solving problems than bitching about them, he suggested I write a hook to prevent myself from missing files. Here it is:

[hooks]
precommit = hg status | (! grep '^?')

Add that to your .hgrc file (and setup some good global hgignore rules) and you’re golden.