I whipped up a dead-simple greasemonkey script today that increases the size of the comment box when you click on it, and then shrinks it back down again when you click elsewhere.

• Download: GrowReviewComment

In response to the comments on my previous entry about Tackling the Release Engineering:Future queue and a number of conversations I’ve had out-of-band over the past few months, the release engineering team has decided to do away with the Future component. We will be merging all of the bugs therein back into the regular Release Engineering component.

Aside from the simplicity inherent in having fewer components to worry about, there were three main reasons we chose to do this:

1. No other group in the company/community has an explicit “Future” component where they can hibernate bugs.
2. People outside the RelEng team find the Future component opaque, and despite reassurances to the contrary, they often just assume bugs in the Future component won’t be fixed.
3. Moving bugs between the regular Release Engineering component and the Future component makes it very hard to extract meaningful statistics about how our bugs change over time (new bug rate, fix rate, etc.). This makes it hard to change our processes based on data.

What does this mean for you? Nothing, hopefully. I’m planning to move all the bugs out of the Future component on the morning (EST) of Monday, February 15th. People in North America will be on holiday, so this should be a relatively low-traffic time to make this change. My apologies in advance for the RelEng mail bomb you’ll be getting on Tuesday morning. :/

We will still continue to triage bugs using our three bucket approach, only the location of the third bucket will have changed. Our three buckets of bugs correspond to the following:

1. Untriaged: – state: NEW, assigned to: nobody, priority: none (–)
2. Triaged, assigned – state: NEW (we’re not working on it yet) or ASSIGNED (we’re working on it), assigned to: a specific release engineer, priority: P1-3
3. Triaged, unassigned (was Future) – state: NEW, assigned to: nobody, priority: P3-5

In the interest of further simplification, the RelEng team will eschew the Severity field as we move forward, although we won’t change any information already contained in that field if it exists. Instead, we’ll start using the Priority field more diligently. All bugs that have been triaged will have a Priority associated with them from here on out.

For reference, the various Priority levels mean the following to us:

P1: blocker, all other work on hold until this is fixed
P2: being actively worked on
P3: acknowledged as an issue, but not being actively worked on
P4: (future) goals that are either blocked on other work, or that simply have not been started yet
P5: enhancement

There was some concern within the RelEng team that developers might assume that bugs moved out of the Future component were now being worked on. I have faith that people will correctly interpret the implications of their bug still being assigned to nobody@mozilla.org as an indication that, no, their bug is still on the shelf.

Please note that this does not change the triage work that still needs to happen on these once-and-Future bugs, as described in the previous post. We’ve already made some good headway on triage this week, though. The number of “Future” bugs is now down to 304 from a high of 343. Kudos to the people who made that happen. Let’s keep up that great work.

I’m certain most developers first response to having their bug moved to the Future queue is, “Oh, look, my bug has fallen down a well.” Historically speaking, that may not be far from the truth.

What goes in Future?

Why does the Future component make people get punchy? For a long time, the Future component has lacked a decent description, so developers don’t know what it means when their bugs are moved there. Many have started hording their gigawatts in anticipation.

Bugzilla currently has the following small description of the Release Engineering:Future component:

“For longer term projects that have been agreed should be done, but have no immediate plans to so. These are not be part of the regular recurring triage. Advanced planning and placeholder goals for next quarter also go here.”

Despite the grammatical errors, this description is mostly accurate, but what does it actually mean:

• Triaged bugs with no immediate owners go here.
• Sadly, most enhancement bugs end up here unless they will make the core release process better, more streamlined, etc. quickly.
• Bugs that are blocked on longer term projects in other groups go here until there is something for RelEng to do.
• Advanced planning and placeholder goals for next quarter (or later) also go here. That’s pretty self-explanatory.

Remember: this description is for the Future component ONLY. The RelEng team continues to pick up and work on bugs that need to get done on a daily basis.

I’m thinking about how to improve this description, and will get the description updated in Bugzilla once I have achieved some rough consensus. In the meantime, I’ve posted this description of the Future component in the wiki as an ongoing reference.

The (Ever-Increasing) Numbers

At the time of writing, the Release Engineering:Future component has 343 bugs in it. This number has grown steadily over the past year despite having more release engineers on staff, and having made a great many improvements to our release automation and our continuous integration infrastructure.

Our turnaround time for bugs in the Future component is also not stellar. At our urging, the Mozilla Metrics team recently started setting up a dashboard to give us various statistics on our Bugzilla usage. Bugs don’t get fixed in the Future queue, so it’s hard to make truly accurate assessments here, but there are a lot of aging bugs in there. According to the numbers, fully 50% of the bugs in the Future component are older than 1 month and more than 25% are older than 6 months. How this compares to other teams or areas, I can’t say, but it certainly makes me empathize with developers who feel their Future-ed bugs have gone walkabout.

If it makes you feel better in a sadistic way, the majority of bugs filed by RelEng team members go directly into the Future queue. We’re not overly happy about it either.

I have a way? Is that better than a plan?

Things are getting done, though. For example, over the past month, the number of non-Future, release engineering bugs that are actively being worked on has gone from a low water mark of 130 up to over 200 today. For comparison, over the same period, the total number of bugs in the Future queue has slowly crept up from a low of 302 to 323. Mozilla is growing along so many axes that sometimes it feels like keeping the increase in the number of Future bugs to a linear relationship rather than exponential one is an accomplishment in itself.

So how do we stop the increase and start wrestling the Future component back under control?

The first step is triage. Starting this Thursday, February 11th, the RelEng team is going to have bi-weekly triage meetings to specifically prune down the Future queue.

As part of the triage, we’re going to be touching every bug and updating the whiteboard field with searchable tags. Our goal here is to make it easy to find classes of bugs in the Future queue so that duplicates and overlap can be easily eliminated, and fixes can be batched as much as possible.

We’ll be keeping a list of the tags we’re using in the wiki in case you want to follow along.

There are some classes of bugs that we won’t be able to eliminate (e.g. future goals), but hopefully within a few months, we’ll have the Future queue back under control.

It won’t be a quick fix, but it’s one we’re committed to.

Since we implemented the RelEng sheriff (or “buildduty” as it is more informally called) last spring, developers have been getting better about using the buildduty person as the first point of contact for RelEng issues. I’d like to implore people to continue doing so. There are a few exceptions, of course:

• It is outside of regular work hours and/or the designated buildduty person is not available; or,
• you are already working on a bug with a specific release engineer.

We recognize that different release engineers have different core competencies, and it may be tempting to go “straight to the source,” but please respect our process. The process is in place to limit (as much as possible) the interrupt-driven nature of release engineering work and to minimize the impact of those interrupts on ongoing projects. In many cases, the person with the expertise you need will still end up helping you (we’re all in the same IRC channels, after all), but the buildduty person is specifically committed to helping you, even if only as a thin scheduling proxy for another release engineer.

Ben’s post had a good list things that RelEng can help developers with. Based on actual requests over the past few months, here are a few other services we can offer:

• clone a virtual machine for one-off debugging/testing: Sometimes patches work on the try server but (for whatever reason) don’t once checked in. Sometimes tests fail or are intermittently orange. If you’re seeing issues like these after landing a patch, or need access to a specific platform for short-term work, the RelEng team can clone one of our reference machines and give you access (under certain circumstances).
• grab build/extra logs from a build machine (provided they exist). This is often useful if a try server build generates extra, non-standard output

In both the above cases, the best thing to do is talk to the person on buildduty and then file a bug.

Conspicuously missing from Ben’s post was a list of what RelEng can’t do for you. While we can usually point you in the right direction, there are some things that we explicitly cannot do:

• help you debug issues with your own local builds. We’re not (necessarily) build config experts, so unless you’re running with exactly the same configuration as our build machines and don’t have any code patches applied, you’d be better served asking the talented folk in #developers or posting to the newsgroups (m.d.builds or m.d.a.firefox)
• shepherd your patch through the landing process. Standard patch landing rules apply: *you* are expected to be around when you land patches. If builds/tests fail and you’re not around, your patch will likely be backed out.

Need help finding the RelEng sheriff? Here’s the RelEng sheriff schedule.

On the house front, things are obviously progressing well if we’re at the model stage. Kris has a post up over at her blog about some of the green certification systems we’ve been examining and synthesizing (and by “we,” I of course mean “her”) over the past few months. In redux, we’re much more interested in a house that performs well over the long-term in terms of energy savings and being comfortable rather than one-time, “hooray-for-us” points. As such, we’re heavily invested in making design choices and choosing systems that will last.

Our focus will remain the same: look for large-ish sections of RelEng-related project work, whether existing or new, and drive them to completion. Some things we’ve worked on previously include properly integrating l10n builds into our buildbot infrastructure, and enabling nightly updates for localized builds. We are currently working on standardizing on one set of update tools and getting localized builds put together for mobile.

John has affectionately dubbed our sub-group “Big Chunks” in reference to the scope of work we take on. That’s a terrible name, and that’s where you come in. I’m soliciting alternate naming ideas for our new group. The leading candidate so far was proposed by Armen: “Chunk Norris”. I have faith in the collective wisdom of the internet to come up with something better (perhaps not even referencing “chunks” at all) or at the very least to entertain me with your attempts.

And….GO!

I’ve done my share of releases over the past few quarters, and until the current release (3.5.4, still in progress as of writing), I would have said that I understood the process pretty well.

The RelEng team has a convention when documenting our release procedures. When a given step completes successfully, we mark it with ‘No problems’ in the build notes. The problem with ‘No problems’ is that it isn’t instructive.

The current 3.5.4 release, for which I’m responsible on the RelEng side, has been about as far from ‘No problems’ as I can reasonably imagine. The in-progress build notes are available here for the rubberneckers and/or morbidly curious. We’ve had respins, we’ve had aborted betas, we’ve had automation interrupted at the very first step by tagging collisions, we’ve had single-locale rebuilds. I’ve cursed each and every one of these things as they’ve happened, but I’ve muddled through them with the help of others on my team. In the process, I’ve learned far more about the underlying processes than I ever would have if our release automation had simply chugged along to completion.

It turns out what I understood about our release procedure was how to follow a recipe. As long as I didn’t run out of ingredients and had all the right pots and pans, the recipe came out just as expected. Fortunately (and yes, I am going to say fortunately here because that’s the whole point), this current release has enabled me to ad-lib in the release kitchen a lot more when things are not perfect. Now if only I could find the release automation equivalent of bacon so I can declare culinary victory much earlier in the release process…but perhaps that’s where this metaphor breaks down.

Don’t get me wrong, every time I do a release, I’ll still be praying to anyone that will listen for build steps to complete with ‘No problems,’ but I no longer fear the occasional and inevitable problems that will crop up in a system of this complexity.

Kids make great straight men, so after quickly explaining the rules of the game, in no time at all I had Will “two-ing” the sandbox, and so on. My son’s laugh is one of my most favorite things in the world. Hearing him explode with laughter when he realized that he had just claimed to have eaten the sandbox made my whole week.

After we ran through the sandbox game a few more times (making sure that Daddy “eight” a few sandboxes for parity) I took him to the computer to show him the source material. As is always the way with Sesame Street clips on YouTube, one clip invariably led to another, as we wistfully plumbed the depths of my childhood, sharing deep thoughts on the proper way to check if people are actually asleep and pondering metaphysical questions such as how Ernie is able to pull a hamburger out of a TV.

I often take this always-on, instant access for granted. There are plenty of mundane aspects to the web. I now *expect* my bank, government, toy store, etc… to offer their services to me online as a matter of convenience, and I will often not patronize them if they don’t. Well, except for the government. If only it was that easy!

I am also amazed by the enormous creativity I encounter via the web every day. Websites like thesixtyone have found a way to mash up music with brownie points that has me interested in a variety of music that I haven’t cared about since high school. I find new podcasts and comics every week that make me ask: “If something like this has been out there for *this* long and I’m only finding out about it now, how much more am I missing?”

Well, open web, here’s the deal: you keep creating stuff, I’ll keep doing my part. Much love.

I usually like MovieBob’s reviews over at the Escapist, but before actually seeing G.I. Joe: The Rise of Cobra, I just assumed he had lost the thread when he posted a positive review of the movie given the overwhelming negativity of others. Having now seen the film, I think he’s absolutely spot on.

This is the type of fun summer movie that I haven’t seen in ages: a decent plot that is coherent and doesn’t expect me to check my brain at the door, larger-than-life characters in over-the-top action sequences, and enough cool tech for 2 regular action movies. If someone had just thought to take the lips off of Snake Eyes’ costume, I dare say it would have been damn-near perfect*.

* “Perfect” in the summer movie sense. This movie won’t be winning any Oscars, let’s not kid ourselves here.