No problems

Posted 10 months, 2 weeks ago at 11:05. 1 comment

The Mozilla release engineering (RelEng) team has grown substantially over the past two years. Some members of the team have certain domain-specific focuses (e.g. talos, mobile), but one of our primary team goals has been to get our release automation to the point where anyone on our team can handle a release. Given the emerging branch picture for Mozilla code where we might see simultaneous releases on 3+ code branches (a.k.a release-apalooza), this is becoming increasingly important.

I’ve done my share of releases over the past few quarters, and until the current release (3.5.4, still in progress as of writing), I would have said that I understood the process pretty well.

The RelEng team has a convention when documenting our release procedures. When a given step completes successfully, we mark it with ‘No problems’ in the build notes. The problem with ‘No problems’ is that it isn’t instructive.

The current 3.5.4 release, for which I’m responsible on the RelEng side, has been about as far from ‘No problems’ as I can reasonably imagine. The in-progress build notes are available here for the rubberneckers and/or morbidly curious. We’ve had respins, we’ve had aborted betas, we’ve had automation interrupted at the very first step by tagging collisions, we’ve had single-locale rebuilds. I’ve cursed each and every one of these things as they’ve happened, but I’ve muddled through them with the help of others on my team. In the process, I’ve learned far more about the underlying processes than I ever would have if our release automation had simply chugged along to completion.

It turns out what I understood about our release procedure was how to follow a recipe. As long as I didn’t run out of ingredients and had all the right pots and pans, the recipe came out just as expected. Fortunately (and yes, I am going to say fortunately here because that’s the whole point), this current release has enabled me to ad-lib in the release kitchen a lot more when things are not perfect. Now if only I could find the release automation equivalent of bacon so I can declare culinary victory much earlier in the release process…but perhaps that’s where this metaphor breaks down.

Don’t get me wrong, every time I do a release, I’ll still be praying to anyone that will listen for build steps to complete with ‘No problems,’ but I no longer fear the occasional and inevitable problems that will crop up in a system of this complexity.

Current Tunes: Jaytech Music Podcast 022 | Filed under Build/Release, Mozilla |