The changing face of buildduty

Posted 1 month ago at 13:22. 1 comment

Buildduty is the friendly face of Mozilla release engineering that contributors see first. Whether you need a production machine to debug a failure, your try server push is slow, or hg is on the fritz, we’re the ones who dig in, help, and find out why. The buildduty role is almost entirely operational and interrupt-driven: we respond to requests as they come in.

We think it’s important for everyone in release engineering to rotate through the role so they can see how the various systems interact (and fail!) in production. It also allows them to forge the relationships with other teams — developers, sheriffs, release management, developer services, IT — necessary to fix failures when the occur. The challenge has been finding a suitable tenure for buildduty that allows us to make quantifiable improvements in the process.
Continue Reading…

Current Tunes: Chaka Khan - My Love Is Alive | Filed under Build/Release, Mozilla

Better releng patch contribution workflow

Posted 1 month, 3 weeks ago at 11:42. 0 comments

Releng systems are complex, with many moving parts. Making changes to these systems used to be fraught with peril. Even for people who had been working with buildbot for many years, it was hard to write a patch and be certain that wouldn’t break some downstream system.
Continue Reading…

Current Tunes: Nothing | Filed under Build/Release, Firefox, Mozilla, Software

10.8 testing disabled by default on Try

Posted 6 months, 1 week ago at 16:25. 1 comment

Mountain LionIf you’ve recently submitted patches to the Mozilla Try server, you may have been dismayed by the turnaround time for your test results. Indeed, last week we had reports from some developers that they were waiting more than 24 hours to get results for a single Try push in the face of backlogs caused by tree closures.

The chief culprit here was Mountain Lion, or OS X 10.8, which is our smallest pool (99) of test machines. It was not uncommon for there to be over 2,000 pending test jobs for Mountain Lion at any given time last week. Once we reach a pending count that high, we cannot make headway until the weekend when check-in volume drops substantially.

In the face of these delays, developers started landing some patches on mozilla-inbound before the corresponding jobs had finished on Try, and worse still, not killing the obsolete pending jobs on Try. That’s just bad hygiene and practice. Sheriffs had to actively look for the duplicate jobs and kill them up to help decrease load.

We cannot easily increase the size of the Mountain Lion pool. Apple does not allow you to install older OS X versions on new hardware, so our pool size here is capped at the number of machines we bought when 10.8 was released over 2 years ago or what we can scrounge from resellers.

To improve the situation, we made the decision this week to disable 10.8 testing by default on Try. Developers must now select 10.8 explicitly from the “Restrict tests to platform(s)” list on TryChooser if they want to run Mountain Lion tests. If you have an existing Mac Try build that you’d like to back-fill with 10.8 results, please ping the sheriff on duty (sheriffduty) in #developers or #releng and they can help you out *without* incurring another full Try run.

Please note that we do plan to stand up Yosemite (10.10) testing as a replacement for Mountain Lion early in 2015. This is a stop-gap measure until we’re able to do so.

Current Tunes: Nothing | Filed under Build/Release, Firefox, Mozilla

Dispatches from the releng team week: Portland

Posted 1 year ago at 17:11. 1 comment

UntitledReleng has been much more diligent during our current team week about preparing presentations and, more importantly, recording sessions for posterity.

Sessions are still ongoing, but the list of presentations is in the wiki. We will continue to add links there.

Special thanks to Armen for helping remoties get dialed-in and for getting everything recorded.

Current Tunes: Nothing | Filed under Build/Release, Firefox, Meeting Notes, Mozilla, Open Web

If I had a million dollars

Posted 1 year ago at 15:16. 4 comments

Kraft DinnerArmen has a blog post up about the cost savings Mozilla has been able to realize in its continuous integration infrastructure in Amazon over just the last 3 months. This has been a bit of a sea change for release engineering, who have historically been conservative with regards to changing core infrastructure and practices. We’re all coming to grips with the new world order, but I’m quite excited about the possibilities.

Some quick back-of-the-envelope calculations based on other recent numbers from Armen:

  • starting with a low-ball estimate of 7,000 pushes/month, if we project the rate of spending from December ($19/push) over an entire year, we end up with $1,596,000.
  • at the new rate ($6/push), a year of AWS time will cost only $504,000.
  • that’s a yearly savings of $1,092,000.

If history has taught us anything, continued growth will eat in to at least part of that savings, but think of what Mozilla could do with an extra million dollars. Depending on where we hire them, that money could easily buy 5-10 more engineers to continue driving the mission forward.

Current Tunes: Nothing | Filed under Build/Release, Firefox, Mozilla, Open Web, Software

Release engineering holiday coverage

Posted 1 year, 4 months ago at 19:47. 1 comment

Mozilla is graciously giving its staff a two-week break over the holidays. What does this mean for service groups like release engineering?

We too will be away spending time with our families, but we will also continue to monitor the general health of the continuous integration infrastructure. If something happens that knocks an entire platform or datacenter offline, we will stand things back up, but we won’t be worrying about regular day-to-day issues like rebooting individual slaves or fulfilling loan requests.

Barring a chemspill, we also won’t be producing releases over the two-week span, with the exception of Nightly and Aurora daily releases which should continue to happily hum along.

Mozilla staff know how to get in connect with release engineering should it become necessary. For other Mozillians, your best bet is to contact us in the #releng channel on IRC. Ping coop (that’s me), catlee, or hwine if necessary, but please be patient. There may be eggnog involved.

SantaMoz

Current Tunes: Jaytech Music Podcast 071 | Filed under Build/Release, Firefox, Mozilla, Software

On John Leaving Mozilla

Posted 1 year, 4 months ago at 00:56. 2 comments

JägerbombJohn has posted his farewell over on his blog.

John’s been managing me now for 6.5 years. I’ve had other friends/mentors/managers before, but it was under John that I became a manager myself.

Everything I’ve learned about people-first, no-nonsense management has come from him. Everything I’ve learned about cross-group co-ordination and asking the right questions has come from him. Everything I’ve learned about Irish whiskey has come from him.

On the technical- and pure getting-shit-done-side, the Google Tech Talk he gave on “Release Engineering as a Force Multiplier” is probably the only resume he’ll ever need, even if he doesn’t actually need it.

Where does John’s departure leave Mozilla, and more specifically, our release engineering team, and even more specifically, me?

I don’t know, but I’m optimistic, mostly because of the imprint left by John on all of us.

Current Tunes: Audien - Wayfarer | Filed under Build/Release, Firefox, Friends, Management, Mozilla, Nostalgia

Dispatches from the releng team week: Friday

Posted 1 year, 5 months ago at 11:55. 0 comments

All good things must come to an end.
Continue Reading…

Current Tunes: Andrew Bayer - Opening Act | Filed under Build/Release, Firefox, Mozilla, Open Web

Dispatches from the releng team week: Thursday

Posted 1 year, 5 months ago at 18:34. 0 comments

Thursday was a planned outing day, with all of releng decamping to the New England Aquarium for the morning. As a trained marine biologist, it was nice for me to be able to drop some science on my Mozilla colleagues.
Continue Reading…

Current Tunes: Nothing | Filed under Build/Release, Firefox, Mozilla, Open Web

Dispatches from the releng team week: Wednesday

Posted 1 year, 5 months ago at 17:54. 0 comments

In addition to starting to train some of the new hires in the ancient arts of buildduty and releaseduty, we had 3 great sessions today:
Continue Reading…

Current Tunes: Nothing | Filed under Build/Release, Firefox, Mozilla, Open Web