Mac build farm standardization headaches: RFC
Posted 2 years ago at 11:42. 9 comments
The Mozilla build team has had some good success using VMware to manage our build reference platforms for both Windows and Linux. This success has encouraged us to try to find a way to standardize our Mac tinderboxen as well. However, Apple has a heavy investment in hardware, so they haven’t exactly embraced virtualization with open arms. What’s the exact opposite of embrace? 
Our lofty goal is to get all our Mac tinderboxen into an identical config state. This past summer, we chose a baseline install for the Macs as 10.4.7, based largely on the build requirements at the time, and what the majority of our existing tinderbox PPCs were already running. Sadly, this proved not to be specific enough. As IT re-installed more Macs for us up to 10.4.7, it became obvious that we had multiple Darwin kernels underlying the 10.4.7 designation. Our Mac tinderboxen are now split between the original installs running Darwin 8.7.0 (Build 8J135), and the machines reinstalled by IT running Darwin 8.7.2 (Build 8K1079), both of which bill themselves as 10.4.7. IT tells me that they were unable to upgrade/reinstall these machines to the identical Build version due to serial number reuse issues. I’m inclined to believe them, given that they do this kind of thing for a living. There also does not seem to be a magic software update that will take a 8.7.0 build up to 8.7.2, or at least that is my conclusion after spending most of yesterday looking for such a beast. More low-level meta-info for these updates would not be amiss, if anyone from Apple happens to be listening.
All this to say that we are currently stuck with a mish-mash of Darwin kernel versions on our Mac tinderboxen. Perhaps understandably, this makes developers antsy because they can’t be sure the toolchain, etc. is identical. I won’t even bother addressing the problem of the various Mac desktop boxes building Camino. IT just flat-out refuses to touch those, and I don’t blame them. We have also had several new Intel Xserves on order for…well, ever that will need to be integrated into the mix once Apple actually releases them.
Finding the right install version and kernel to standardize on is only part of the problem though. After these Macs are reinstalled, there is still a raft of updates and ancillary software that needs to be installed — Xcode, Java, and more — to fully realize the Mac ref platform. We already have an internal Mac server setup to stage software updates from Apple so we can pick and choose what gets presented to our build farm machines, but that still means running Software Update on each Mac after an install. What we’d love to move to is something akin to the VM solution we’re using for Windows and Linux where the OS and all the necessary updates and software can coexist on a single image ready for install.
TR has suggested Carbon Copy Cloner, but AFAICT, this is meant for a single install backup scenario, and I’m not sure it would cope with things like needing to change the serial number for each machine and such. I’m happy to be proved wrong in this regard though.
Does anyone in the community have any experience maintaining a collection of Mac machines (build farm or otherwise) in an identical configuration, and can recommend ghosting software or process solutions for doing so?
Update: NetRestore, rather than CCC, seems to be the product of choice for this. Bonus points for also being open source!
Carbon Copy Cloner. Read the website, it does what you want. I used to maintain computer labs with dozens of Macs with different hardware (G4 and G5). You just make your image on your most advanced Mac (to make sure all the drivers you need are there), and then use Carbon Copy Cloner to put that image on all your other machines. It really works like a charm.
I am not sure I understand your serial number problem, though.
You may want to look into netbooting from a network disk image (or net restores). That’s a very common setup for labs.
Various releases of 10.4.7 exist for hardware enablement, to support new systems that the original 10.4.7 didn’t support.
I _believe_ (not sure) that it would be a good idea to move to 10.4.8 / Xcode 2.4.1 . My understanding is that it makes it simpler to prepare for Leopard compatibility.
Re: CCC - two problems here that I can see:
1) Lack of Universal Binary - not a showstopper, because honestly, how often are you going to be cloning disks, but still a performance hit for running under Rosetta; and,
2) I have read the website, and have not found mention of how serial number munging is handled in the mass deployment scenario. This will break the interaction with management server for admin, updates, etc. that we have now.
In fact, after delving deeper into the CCC forums, it would seem that NetRestore is the product that we really should be considering. This would address Eric’s comment as well.
Opposite of embrace?
“Exbrace,” of course.
At least when apple starts shipping virtualization you should (hopefully) be able to run different Mac OS X versions in the same box
Hey Coop,
I was just flipping through some news items and found this:
http://www.faronics.com/html/DFMac.asp
It might do what you want and is a UB. Don’t know if it’ll manage serial number munging or not.
As for the opposite of “embrace”, it’s clearly “shoo”.
You can try a ‘freed’ kernel in order to be able to use a virtual machine:
http://semthex.freeflux.net/nebukadnezar-faq/
This allows the installation on VmWare et al, though i don’t know if the modifications make it invalid for your purposes.