Tackling the Release Engineering:Future queue
Posted 1 month, 1 week ago at 17:22. 4 comments
The Release Engineering:Future bugzilla component alternately inspires feelings of sadness, loathing, and contempt…and that’s just within the RelEng team!
I’m certain most developers first response to having their bug moved to the Future queue is, “Oh, look, my bug has fallen down a well.” Historically speaking, that may not be far from the truth.
What goes in Future?
Why does the Future component make people get punchy? For a long time, the Future component has lacked a decent description, so developers don’t know what it means when their bugs are moved there. Many have started hording their gigawatts in anticipation.
Bugzilla currently has the following small description of the Release Engineering:Future component:
“For longer term projects that have been agreed should be done, but have no immediate plans to so. These are not be part of the regular recurring triage. Advanced planning and placeholder goals for next quarter also go here.”
Despite the grammatical errors, this description is mostly accurate, but what does it actually mean:
- Triaged bugs with no immediate owners go here.
- Sadly, most enhancement bugs end up here unless they will make the core release process better, more streamlined, etc. quickly.
- Bugs that are blocked on longer term projects in other groups go here until there is something for RelEng to do.
- Advanced planning and placeholder goals for next quarter (or later) also go here. That’s pretty self-explanatory.
Remember: this description is for the Future component ONLY. The RelEng team continues to pick up and work on bugs that need to get done on a daily basis.
I’m thinking about how to improve this description, and will get the description updated in Bugzilla once I have achieved some rough consensus. In the meantime, I’ve posted this description of the Future component in the wiki as an ongoing reference.
The (Ever-Increasing) Numbers
At the time of writing, the Release Engineering:Future component has 343 bugs in it. This number has grown steadily over the past year despite having more release engineers on staff, and having made a great many improvements to our release automation and our continuous integration infrastructure.
Our turnaround time for bugs in the Future component is also not stellar. At our urging, the Mozilla Metrics team recently started setting up a dashboard to give us various statistics on our Bugzilla usage. Bugs don’t get fixed in the Future queue, so it’s hard to make truly accurate assessments here, but there are a lot of aging bugs in there. According to the numbers, fully 50% of the bugs in the Future component are older than 1 month and more than 25% are older than 6 months. How this compares to other teams or areas, I can’t say, but it certainly makes me empathize with developers who feel their Future-ed bugs have gone walkabout.
If it makes you feel better in a sadistic way, the majority of bugs filed by RelEng team members go directly into the Future queue. We’re not overly happy about it either. 
I have a way? Is that better than a plan?
Things are getting done, though. For example, over the past month, the number of non-Future, release engineering bugs that are actively being worked on has gone from a low water mark of 130 up to over 200 today. For comparison, over the same period, the total number of bugs in the Future queue has slowly crept up from a low of 302 to 323. Mozilla is growing along so many axes that sometimes it feels like keeping the increase in the number of Future bugs to a linear relationship rather than exponential one is an accomplishment in itself.
So how do we stop the increase and start wrestling the Future component back under control?
The first step is triage. Starting this Thursday, February 11th, the RelEng team is going to have bi-weekly triage meetings to specifically prune down the Future queue.
As part of the triage, we’re going to be touching every bug and updating the whiteboard field with searchable tags. Our goal here is to make it easy to find classes of bugs in the Future queue so that duplicates and overlap can be easily eliminated, and fixes can be batched as much as possible.
We’ll be keeping a list of the tags we’re using in the wiki in case you want to follow along.
There are some classes of bugs that we won’t be able to eliminate (e.g. future goals), but hopefully within a few months, we’ll have the Future queue back under control.
It won’t be a quick fix, but it’s one we’re committed to.
Coop: this is great, and will certainly help people set their expectations correctly. Would it be possible to merge the RE and RE:F components, and use the tags you describe to distinguish triaged/assigned state? That would bring it more in line with how most bugs in Bugzilla are managed, and I think generally reduce the amount of anxiety that people feel when their request is triaged.
What shaver said, for one, besides that:
How long do bugs take to get a patch, how long for the config to work, and how long in staging? At least for the bugmail I get, config and staging seem to be more of a drain than hoped?
How many external contributors does releng have? Are there any “good first bugs” for people to get started?
> Would it be possible to merge the RE and RE:F components, and use the tags you describe to distinguish triaged/assigned state? That would bring it more in line with how most bugs in Bugzilla are managed, and I think generally reduce the amount of anxiety that people feel when their request is triaged.
I do feel your pain here. I’ve lobbied for a while to get us back (because we did used to track things this way) to a single pool of bugs to bring us more in line with the rest of the organization/community. The current setup makes tracking turnaround time and such almost impossible.
I’m not sure where the resistance to switching back is coming from within the group. Most of us enjoy having smaller bug lists, but the unfortunate side effect is that bugs that wind up in the Future queue are forgotten…out-of-sight, out-of-mind.
> How long do bugs take to get a patch, how long for the config to work, and how long in staging? At least for the bugmail I get, config and staging seem to be more of a drain than hoped?
Again, it’s almost impossible to know under the current system without doing math that bridges the flip-flopping of components. I can put in a call to Metrics to see how difficult this would be. We should be able to get some rudimentary data based simply on bug creation time.
As to staging, the staging environment is truly a boon and we could not operate without it. Unfortunately, as such, there is always a *lot* of contention to use the staging env by members of the release engineering team. We never have enough staging machines to satisfy everyone at once, and the pressure to put new machines directly into production to lessen wait times is huge. The whole team ends up waiting on staging. Maybe when we mothball the next batch of minis, we can give 3 to each release engineer for their own private staging setup.
> How many external contributors does releng have? Are there any “good first bugs” for people to get started?
Does rhelmer count?
A “good first bug” will depend a lot on your background, but the only ones that are really accessible to outsiders are the buildbot ones, and even then, you need to have a rudimentary master/slave setup to test any changes. Maybe we need instructions on setting that up somewhere.