Reviving BuildFaster: Measuring progress

Posted 2 years, 2 months ago at 11:39. 3 comments

Pull thread to unravelIn my previous post, I outlined the four main parts in our plan of attack for making the build system faster. But how will we know how well we’re doing, and how will we know when we’re done? How fast is fast enough?

The first step was relatively easy, and involved fixing the data submission from the build automation to brasstacks. Now we can once again see how our average build times are changing on a day-to-day basis.

The graph looks especially nice right now due to turning on pymake on August 30th which netted us a ~25% reduction in build times on Windows (32-bit). (Here is a static image of the graph if you happen to be visiting from the future.) sid0 has a patch in progress to also use pymake for Window 64-bit builds. I look forward to enabling that soon.

The next step is going to take a little more work.

Greg (gps) would like to start tracking CPU utilization as a measure of the efficiency of the build system. This will help address the “how fast is fast enough?” question by providing an initial plateau: get CPU utilization as close to 100% as possible on the various build platforms (how close to 100% is close enough? shhh, stop asking hard questions). As Greg notes,

“…end-to-end build times aren’t going to decrease if the CPU utilization during the build doesn’t go up. The problem today is getting build actions/processes to spawn sooner/faster/on all cores. If we are at 100% CPU utilization on all cores, the problem will be getting them to finish faster. Those are two very different problems.”

One of the tools that Greg has been working on, mach, can report on CPU (and disk I/O) utilization, but it relies on having python 2.7 available on all the build platforms. I’ve picked up bug 602908 to help drive that forward. Having a more recent version of python available will also help other efforts to speed-up end-to-end times, such as nascent work to try running unittests in parallel where possible.

Some progress already made, and lots of work in flight here. Please check out the thread in mozilla.dev.builds where Greg et al. have recently decided to sandboxed Python files for the extracted Makefile data (Part 4 from my previous post).

Onward!

Current Tunes: The Offspring - It'll Be A Long Time | Filed under Build/Release, Firefox, Mozilla, Software |

3 Replies

  1. Very well written!
    Looking forward to see more splendid-ness like this.

  2. Inspired by a post by John O’Duinn, I wrote a script to generate a overview of all the tinderbox tasks. I wasn’t aware that a graph server for that already existed.

    But I also tried to have a look at if the unzip/delete time could be removed. For a mochitest it takes about 10% of the time to unzipping the tools, firefox build, and tests zip files (the tools-folder is currently fetched using a “hg clone”, I assume it might as well use a “wget http://hg.mozilla.org/…/tip.zip”). If I instead mapped the zips using Pismo Mount File (which have a command line interface and can make the zip-files look like directories), the tests still seemed to run (had to add an option to move the websock.log), but it of course eliminated the unzip/delete time (since deleting a large zip-file is fast) and didn’t seem to increase the test time (on my slow machine it cut the test time by ~25%). Maybe it could also be tried for the builds, perhaps even with a ram-disk for the output.


Leave a Reply