Reviving BuildFaster: Plan of attack

Posted 2 years, 5 months ago at 11:13. 3 comments

Pull thread to unravelThe timing of Greg Szorc’s (gps) blog post about Improving Mozilla’s Build System couldn’t have been better.

In his post, gps outlines a 6-part plan for untangling the build system. After some discussion at our first meeting, we decided that while all six parts are useful, the first four parts accomplish our “short-term” goals. The first four parts are:

  1. No Rules – we still need build rules obviously, but they need to live in the correct place. Furthermore, rule collections like rules.mk (and packager.mk) need to DIAF…er, I mean, should be broken up into logical groups of libraries that can be included and reused as required. Ted has filed bug 769378 to get this started.
  2. Eliminate Make File Content Not Related to Building – there is some institutional inertia here because we’ve been telling developers for a while to do things like make -C $(OBJDIR) reftest. There are other things, however, that also should not be in the makefile. gps has filed the following bugs, which should be self-explanatory:
    • Removing all $(shell) invocations from Makefile.in’s (bug 769390)
    • Removing all filesystem functions from Makefile.in’s (bug 769407)
    • Migrating non-buildy aspects of make files elsewhere (bug 769394)
  3. Make File Style Conventions and Enforced Review Requirements – this is a major concern of the Build:Config peers. Development work at Mozilla can move pretty fast some times. This has occasionally led to some project teams pushing forward with their own Makefile work with a focus on getting it done rather than playing nicely with others. We want to be able to move fast, but we also need to commit to cleaning up this work after the fact. We need to get the build system to a better place where the patterns that get copied from the tree hurt us less. We can help this by publishing some best practices, and using some automated tools/checkers written by gps and others.
  4. Extracting Data from Makefiles – moving the data out of Makefiles into json (or similar) allows us to be agnostic about the final build tool. We could continue using make, or trivially move to tup, GYP, or whichever tool individual developers prefer (XCode, Visual Studio, …).

On top of the above, there are two other efforts that we’re pursuing:

  1. Joey will continue to work on his project to derecursify (flatten) the Makefiles. This is akin to Part 5 in gps’ blog post, but provides some initial build speed-ups and doesn’t necessarily need to happen serially.
  2. Switch to using pymake by default on Windows. Most (all?) developers do this already due to the speed-up (up to 30% by some reckoning), so it’s time the Mozilla’s continuous integration system caught up.

Previously – Reviving BuildFaster: Fixing Makefiles

Current Tunes: Above and Beyond - Trance Around The World 433 - 2012-07-13 | Filed under Build/Release, Mozilla |

3 Replies

  1. Awesome. Glad to see some momentum building on this!

  2. Only 30%? Since msys make regularly crashes with parallel builds, a -j12 build with pymake is about 8x as fast as msys make is, for me at least!

  3. We now have https://bugzilla.mozilla.org/show_bug.cgi?id=774049 on file as a master tracking bug.


Leave a Reply