Posted 2 years ago at 11:42. 9 comments
The Mozilla build team has had some good success using VMware to manage our build reference platforms for both Windows and Linux. This success has encouraged us to try to find a way to standardize our Mac tinderboxen as well. However, Apple has a heavy investment in hardware, so they haven’t exactly embraced virtualization with open arms. What’s the exact opposite of embrace? 
Our lofty goal is to get all our Mac tinderboxen into an identical config state. This past summer, we chose a baseline install for the Macs as 10.4.7, based largely on the build requirements at the time, and what the majority of our existing tinderbox PPCs were already running. Sadly, this proved not to be specific enough. As IT re-installed more Macs for us up to 10.4.7, it became obvious that we had multiple Darwin kernels underlying the 10.4.7 designation. Our Mac tinderboxen are now split between the original installs running Darwin 8.7.0 (Build 8J135), and the machines reinstalled by IT running Darwin 8.7.2 (Build 8K1079), both of which bill themselves as 10.4.7. IT tells me that they were unable to upgrade/reinstall these machines to the identical Build version due to serial number reuse issues. I’m inclined to believe them, given that they do this kind of thing for a living. There also does not seem to be a magic software update that will take a 8.7.0 build up to 8.7.2, or at least that is my conclusion after spending most of yesterday looking for such a beast. More low-level meta-info for these updates would not be amiss, if anyone from Apple happens to be listening.
All this to say that we are currently stuck with a mish-mash of Darwin kernel versions on our Mac tinderboxen. Perhaps understandably, this makes developers antsy because they can’t be sure the toolchain, etc. is identical. I won’t even bother addressing the problem of the various Mac desktop boxes building Camino. IT just flat-out refuses to touch those, and I don’t blame them. We have also had several new Intel Xserves on order for…well, ever that will need to be integrated into the mix once Apple actually releases them.
Finding the right install version and kernel to standardize on is only part of the problem though. After these Macs are reinstalled, there is still a raft of updates and ancillary software that needs to be installed — Xcode, Java, and more — to fully realize the Mac ref platform. We already have an internal Mac server setup to stage software updates from Apple so we can pick and choose what gets presented to our build farm machines, but that still means running Software Update on each Mac after an install. What we’d love to move to is something akin to the VM solution we’re using for Windows and Linux where the OS and all the necessary updates and software can coexist on a single image ready for install.
TR has suggested Carbon Copy Cloner, but AFAICT, this is meant for a single install backup scenario, and I’m not sure it would cope with things like needing to change the serial number for each machine and such. I’m happy to be proved wrong in this regard though.
Does anyone in the community have any experience maintaining a collection of Mac machines (build farm or otherwise) in an identical configuration, and can recommend ghosting software or process solutions for doing so?
Update: NetRestore, rather than CCC, seems to be the product of choice for this. Bonus points for also being open source!
Current Tunes: Gorillaz - Fire Coming Out Of The Monkey's Head | Filed under Build/Release, Mozilla
Posted 2 years ago at 00:52. 0 comments
Lots of discussion and decision-making last week, only some of which was fueled by well-deserved booze. Here’s a wrap-up of the salient points as they pertain to me.
Build/Release:
- There are lots of releases planned prior to Christmas, although nothing quite as whiz-bang as Firefox 2. I’ll be helping out wherever I can with the various RCs, alphas, and partner builds, even if it’s just to continue handling the daily flood of tinderbox requests so that others don’t have to.
- Continued work to get our Mac tinderbox “ref” platform implemented across all our current tinderboxen, along with integrating the new Intel xserves into the mix when they arrive. Man, I really wish Apple were making this easier for me, but software updates, in conjunction with having multiple Apple build IDs for the same software version, are really biting my ass right now.
QA:
- davel is leaving Mozilla soon and we wish him well. Much of the unit testing work that he was championing will be taken over by robcee. Again, I’ll be helping out where I can, likely wherever interaction with the build team is required. I already have one foot in each camp anyway.
Litmus:
- There was much talk at the summit about where (specifically, on which products) Mozilla should be focusing its energies. Nothing is set in stone yet, but things will be changing. As our product focus changes, we need to be able to uplift community members to continue testing products that we cannot supply adequate QA resources for. This means implementing product-based admin privileges in Litmus, which was on file but not a priority before.
- Zach and I will be improving the new Litmus install story over the next few months to make it as easy as possible for a 3rd party to install Litmus and use it for their own needs. The aim here is to encourage more people to adopt Litmus as a tool and ease our burden of development. There are lots of other build, release, and QA tools that need writing, so I’d love to get Litmus into maintenance mode ASAP.
- Now that I’ve had my ideas for the test runs interface vetted by the QA team, I can plow ahead and get them done. Hooray for in-person whiteboarding!
Current Tunes: Orbital - Lush 3-4 (Warrior Drift Psychick Warriors Ov Gaia) | Filed under Build/Release, Litmus, Mozilla, QA
Posted 2 years ago at 12:14. 0 comments
I finished migrating tinderbox configs from internal to public CVS late last night. A quick check of the now up-to-date Build:Farm wiki will show you that the overwhelming majority of configs that we care about — core products on one of trunk, 1.8 branch, or 1.8.0 branch — are now in public CVS for the world to see. Hopefully this will lift the curtain on a part of the Mozilla build process that has heretofore been in the hands of only a small number of people.
We invite people to examine the configs. The build team is always willing to review patches if you think something needs updating. The relevant CVS module is mozilla/tools/tinderbox-configs and the various branches can be found in the wiki mentioned above.
As to the remaining builds that still do not have their configs in CVS, if you are a developer who cares about one of these builds and would like to see the config find its way into public CVS, please contact me and we’ll work out a whisky-based payment schedule ;-). This applies also to any builds which I may have inadvertently missed.
Current Tunes: LL Cool J - Big Ole Butt | Filed under Build/Release, Mozilla
Posted 2 years, 3 months ago at 11:50. 0 comments
TR Fullhart joined the Mozilla Build team full-time this week. Given his recent experience with build-related software tools, we’re hoping he can help us get our own software tool ship in better order (among other things).
Welcome!
Current Tunes: Traci Lords - Control | Filed under Build/Release, Mozilla
Posted 2 years, 7 months ago at 11:36. 1 comment
Paul has already posted as much, but I would be remiss if I did not take the time to welcome Rob Helmer myself. Rob has a solid history as a build/release engineer, and adds some much needed bandwidth to the Mozilla build team.
Current Tunes: The Chemical Brothers - Sidewinder (312 Vs 216 Stomp Mix) / Doin' It After Dark (D-Ski's Dance) / Don't Stop The Rock / To A Nation Rockin' | Filed under Build/Release, Mozilla
Posted 2 years, 8 months ago at 14:05. 0 comments
Paul and I appreciate all the help we’ve received over the last little while with regards to Tinderbox maintenance. However, when things go awry, it makes it harder for us to track down why.
If you’re going to be making changes to a Tinderbox config file, can I ask that after you’re changes are made, that you check the syntax of the file before restarting the Tinderbox?
i.e.: perl -c tinder-config.pl
In my experience, Tinderboxes generally fall off the map due to simple syntax errors in the config files, and this is a pretty easy way to avoid that problem before it happens.
Current Tunes: Asian Dub Foundation - Rafi | Filed under Build/Release, Mozilla
Posted 2 years, 9 months ago at 18:35. 0 comments
We put bug 326465 to bed last Friday, and got the Mac builds and updates pushed out Yahoo! shortly thereafter. Thanks to everyone who jumped in to help: bsmedberg, josh, mento, bent, the Mozilla QA team, and, of course, preed.
It ended up being a fairly simple bug — off-by-one level of directory hierarchy for patch generation on Mac — made difficult by a lack of end-to-end knowledge of update patch creation. We now have a much better understanding of how this process works, and better still, who to turn to when we can’t figure out problems with the update system.
I was more than a little bummed that it took as long as it did to get this bug fixed; I wish I had known more about the update process at the outset. However, I was left with a greater appreciation for the Mozilla team/community and the collaborative way in which the bug did eventually end up getting resolved. Thanks again.
Current Tunes: The Chemical Brothers - It began in Afrika, courtesy of Pandora | Filed under Build/Release, Mozilla
Posted 2 years, 9 months ago at 11:55. 1 comment
So, yes, as preed has announced, we’re going to try to keep the #build IRC channel open from now on. We’re trying to be a little more transparent, with the bonus side-effect of removing some build-related chatter from other channels (notably #developers and #bmo).
Please note, however, that “build-related” refers specifically to Mozilla server ops-type requests, e.g. “moz180-win32-tbox is out of space AGAIN,” and not to general “what are the proper mozconfig options to build with foo?”-type questions. We don’t have the bandwidth to handle those sorts of questions, and they are probably better answered in #developers anyway.
Please also note that we’re not huge fans of drive-by bug reports and requests. People who pop into the channel, announce a problem, and then skedaddle will likely not find that problem resolved any time soon. If you’re willing to stick around and help out (in the cases where that is appropriate and possible), you’re much more likely to receive timely help.
Current Tunes: Interpol - Untitled, courtesy of Pandora | Filed under Build/Release, Mozilla
