You are currently browsing the archives for the Build/Release category.
Posted 1 year, 3 months ago at 10:41. 0 comments
The builds populating the XULRunner-Mozilla1.8.0 tinderbox tree are going to be disabled today. To quote bsmedberg from bug 378809:
The 1.8.0 branch is pretty much dead, and all our efforts are focused on the trunk.
As is the way of all things in OSS, that code is still going to exist, we’re just not going to be building it in tinderbox any more, or actively supporting those builds. People are encouraged to investigate the XULRunner trunk instead.
Current Tunes: DJ Gregory - Essential Mix - 2003-11-30 | Filed under Build/Release, Mozilla
Posted 1 year, 3 months ago at 10:25. 0 comments
The Mozilla build farm grew rather organically at the outset. We ended up with a lot of legacy tinderbox systems that were all installed and configured differently. This was especially true for Linux, which can be installed on just about any piece of hardware.
The tinderbox code requires an X server of some sort running on Linux so that it can fire up a new build and run basic tests on it, but there are almost as many ways setup to do this on the Linux tinderboxen as there are machines themselves.
Yesterday, we moved balsa-18branch, a Redhat 7.2 VM, to a new VM host, but attempts to restart the X server as the build user failed with a console ownership error: PAM authentication failed, cannot start X server.
This mailing list post got me most of the way there, but the proper PATH to the file that needed to be touch-ed was actually /var/run/console/USERNAME.
Maybe this will help someone else out, but really, I’m just recording it here so I don’t forget it myself next time I need to restart that VM.
Current Tunes: Trespassers William - It's Been A Shame | Filed under Build/Release, Mozilla
Posted 1 year, 3 months ago at 14:23. 0 comments
We managed to get 4 of the 6 community VMs onto the new host yesterday before mrz and I succumbed to our respective illnesses. We’re going to resume moving the two remaining VMs, sea-linux-tbox and sb-linux-tbox, on Friday when we’re both (hopefully) feeling better.
At that point, it will be up to me to get those machines sanitized for handover to various project admins. There’s still some churn happening with regards to CVS access and such, so this likely won’t happen until sometime next week at the earliest.
Current Tunes: New Order - Blue Monday (Hardfloor Mix Edit) | Filed under Build/Release, Mozilla
Posted 1 year, 3 months ago at 17:00. 1 comment
As a quick synopsis for those not directly involved in what has been going on, the Mozilla build team (with lots of help from the IT team) is moving all of the tinderboxen for community project builds — Camino, SeaMonkey, and Sunbird/Lightning, — onto a separate build network. Build engineers for those products will then have direct access to those build environments for configuration maintenance and debugging. This will resolve a longstanding complaint from community developers about turnaround time for tinderbox requests by allowing/forcing them to fix things themselves. Be careful what you wish for, people! 
Honestly though, we think this will be a boon for everyone. Yes, it will hopefully mean less time spent mucking with random tinderboxen for us, but it will also allow community projects to determine their own build system destiny, while still being able to take advantage of the existing Mozilla build infrastructure for things like automated configuration updates if they so choose.
And now, the update: mrz is going to start migrating VMs used by Mozilla community projects onto their own VM host tomorrow (Wednesday). We’re hoping to bang them all out in one go, but we’ll continue Thursday and Friday evening if necessary. The practical fallout from this is that Mozilla-hosted Linux and Windows builds for community projects may/will have some downtime tomorrow, but it will be after the nightlies are generated and any given build should only be offline for a few hours at most.
Current Tunes: Jimmy Smith & Lyrics Reborn - Stay Loose (Lyrics Reborn Remix) | Filed under Build/Release, Mozilla
Posted 1 year, 5 months ago at 17:26. 2 comments
As of today, the Mozilla build team is going to be handing off all front-line tinderbox support to the Mozilla IT team.
The Why:
The Tinderbox build farm is still the core of our developer infrastructure, providing developers with build access to platforms they might not otherwise have. Sadly, we don’t always pay sufficient attention to it.
The build team is stretched pretty thin dealing with a release cycle that seems at times perpetual. We have the core releases on two branches, a multitude of partner builds on both branches, preview releases on Trunk, and updates to generate for all of the above. Day-to-day tinderbox maintenance frequently falls through the cracks. Unless a failed tinderbox is in the critical path for that week, it often goes unfixed. In the past, squeaky wheels in #build would sometimes get the grease. This may have created the false expectation that the build team was monitoring these builds in the evenings and over the weekends. This was never the case, and when we kicked boxes at those times, it was because we happened to be around. That mode of operation should fail outside of regular business hours, and is not very sustainable staffing-wise to begin with.
Enter IT. IT is staffed by smart, technical, sysadmin-types who are trained to keep servers up-and-running. Perhaps more importantly, IT already has an established on-call rotation for dealing with important issues quickly outside of the standard workday.
The How:
Over the last few months, the build team has worked with IT to implement nagios monitoring for all our key tinderbox systems. We have divided up the build farm into a handful of service tiers. We are no longer reliant on helpful hints in #build that key tinderboxen have gone down. Tier 1 tinderboxen will receive immediate attention from the IT staff member on-call if they stop building, be it in the middle of the workday or the middle of the night. The service levels will look as follows:
| Tier 1 |
24-hour, on-call support |
| Tier 2 |
Support during business hours which equates to Monday - Friday, non-US hoildays, 9 am to 6 pm PST/PDT |
| Tier 3 |
As time permits (may be multiple days/weeks, depending on request/issue) |
Of course, due to the Tinderbox server architecture, IT won’t be paged by the nagios monitoring until a build falls off the waterfall page. This typically takes 12 hours. There is still the need for vigilance here, especially if the tinderbox you happen to care about isn’t ranked as Tier 1 or isn’t monitored by nagios at all.
For these situations, and indeed for all other tinderbox inquiries going forward, filing bugs in Bugzilla against IT with the appropriate severity is the correct way to proceed. As I’ve mentioned before, IT is very good about managing their bug queue. Your request will be heard.
The build team is not washing its hands of tinderbox. We will be there to help IT with tinderbox as required, but we *will* be taking a step back in order to concentrate on other things: improving the release process, switching version control systems, and such.
It may take a little while to work out the kinks in this new system, but we’re confident that this arrangement will lead to a better tinderbox experience for everybody. We ask that you extend IT the same courtesy you would afford the build team as they learn the tinderbox ropes over the next few months…or you could be nice to them instead.
Current Tunes: Junior Jack and Kid Creme - Essential Mix - 2004/05/23 | Filed under Build/Release, Mozilla
Posted 1 year, 5 months ago at 11:13. 0 comments
preed has already blogged about this, but I would be remiss if I didn’t also mention it here. If you check in code that requires tinderboxes to be clobbered, you can now trigger that clobber yourself. This is true for almost all build machines, the sole exception being extra coverage builds that are not pulling their configs from CVS.
Full details on the process can be found in the wiki.
Current Tunes: The Waterlillies - Tempted | Filed under Build/Release, Mozilla
Posted 1 year, 6 months ago at 15:33. 1 comment
The Mozilla IT team sets the gold standard for managing their bug queue down to near-zero on a daily basis. It’s an admirable accomplishment, and one that the build team would like to replicate.
Ironically, this may actually have the side-effect of making people less happy with us in the short-term.
The build team continues to be pretty busy. That’s not exactly news, but our current bug list belies what we’re actually working on. None of us are actively working on all of the 10 or so bugs that are currently assigned to each of us. It’s part historical artifact and part poor queue management.
One of the ways that IT manages their queue so effectively is by being absolutely ruthless about re-assigning and closing bugs. Bugs don’t stay unassigned, if IT is blocked on someone else outside the team the bug is immediately re-assigned, and bugs waiting on input are closed or WONTFIXed fast if there’s no response.
Our first goal as a team is going to be triage. We’ll start by re-assigning bugs already assigned to individual team members back into the general pool where appropriate. I’m going to do this to my own build bug queue today. This will give everyone a more accurate picture of what is actually in progress.
After that, we’ll get together as a team and triage down the build bug pool. The exact criteria we’ll be using for triage is still under discussion. Where appropriate, we’ll comment in bugs on priorities and time frames, but in many cases, we’ll be swinging a big WONTFIX stick. The end goal is to get the build team to a state where it has never (to my knowledge) been before: able to respond quickly to new requests without distraction.
We’ll get Justin to hold our hands if we get scared.
When *your* bug ends up back in the general pool over the next few weeks, please don’t take it as a slight. However, be prepared to fight for your bug if you think it’s important.
Current Tunes: Journeyman - Latneiro (Woobs Sunrise Dub) | Filed under Build/Release, Mozilla
Posted 1 year, 6 months ago at 09:52. 0 comments
SeaMonkey performed well in the role of guinea pig yesterday, becoming the first Mozilla-based build to be deployed on the new Linux reference VM. The new VM is even generating the nightlies now.
A new VM means a new toolchain. If you have some time to download these builds and give them a spin, the build and SeaMonkey teams would appreciate the extra verification. The nightlies can be found here:
http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/latest-trunk/
If these new builds pan out, it will pave the way for the build team (well, probably me) to begin moving all of our Linux builds over to ref VMs sometime in the “near” future.
Thanks to bsmedberg for his help throughout the day yesterday. With his help (second commit’s the charm), I was able to get the TUnit test running and reporting *properly* for the new SeaMonkey build too.
Current Tunes: Groove Armada - Essential Mix - 2002-10-13 | Filed under Build/Release, Mozilla
Posted 1 year, 8 months ago at 11:50. 0 comments
Never before has so much coffee come out of my nose. Fantastic.
The VCS discussion itself at the summit was much more civil than I was expecting — there was a minimum of spine-ripping — which I believe is a testament to the amount of comparative legwork done by the principals beforehand. Kudos for that.
Current Tunes: Carol Lynn Townes - 99 1/2 | Filed under Build/Release, Mozilla
Posted 1 year, 9 months ago at 11:42. 9 comments
The Mozilla build team has had some good success using VMware to manage our build reference platforms for both Windows and Linux. This success has encouraged us to try to find a way to standardize our Mac tinderboxen as well. However, Apple has a heavy investment in hardware, so they haven’t exactly embraced virtualization with open arms. What’s the exact opposite of embrace?
Our lofty goal is to get all our Mac tinderboxen into an identical config state. This past summer, we chose a baseline install for the Macs as 10.4.7, based largely on the build requirements at the time, and what the majority of our existing tinderbox PPCs were already running. Sadly, this proved not to be specific enough. As IT re-installed more Macs for us up to 10.4.7, it became obvious that we had multiple Darwin kernels underlying the 10.4.7 designation. Our Mac tinderboxen are now split between the original installs running Darwin 8.7.0 (Build 8J135), and the machines reinstalled by IT running Darwin 8.7.2 (Build 8K1079), both of which bill themselves as 10.4.7. IT tells me that they were unable to upgrade/reinstall these machines to the identical Build version due to serial number reuse issues. I’m inclined to believe them, given that they do this kind of thing for a living. There also does not seem to be a magic software update that will take a 8.7.0 build up to 8.7.2, or at least that is my conclusion after spending most of yesterday looking for such a beast. More low-level meta-info for these updates would not be amiss, if anyone from Apple happens to be listening.
All this to say that we are currently stuck with a mish-mash of Darwin kernel versions on our Mac tinderboxen. Perhaps understandably, this makes developers antsy because they can’t be sure the toolchain, etc. is identical. I won’t even bother addressing the problem of the various Mac desktop boxes building Camino. IT just flat-out refuses to touch those, and I don’t blame them. We have also had several new Intel Xserves on order for…well, ever that will need to be integrated into the mix once Apple actually releases them.
Finding the right install version and kernel to standardize on is only part of the problem though. After these Macs are reinstalled, there is still a raft of updates and ancillary software that needs to be installed — Xcode, Java, and more — to fully realize the Mac ref platform. We already have an internal Mac server setup to stage software updates from Apple so we can pick and choose what gets presented to our build farm machines, but that still means running Software Update on each Mac after an install. What we’d love to move to is something akin to the VM solution we’re using for Windows and Linux where the OS and all the necessary updates and software can coexist on a single image ready for install.
TR has suggested Carbon Copy Cloner, but AFAICT, this is meant for a single install backup scenario, and I’m not sure it would cope with things like needing to change the serial number for each machine and such. I’m happy to be proved wrong in this regard though.
Does anyone in the community have any experience maintaining a collection of Mac machines (build farm or otherwise) in an identical configuration, and can recommend ghosting software or process solutions for doing so?
Update: NetRestore, rather than CCC, seems to be the product of choice for this. Bonus points for also being open source!
Current Tunes: Gorillaz - Fire Coming Out Of The Monkey's Head | Filed under Build/Release, Mozilla
