Litmus Downtime tonight, 7pm-9pm PDT
Posted 1 year, 7 months ago at 13:39. 0 comments
Litmus will be offline for a few hours tonight as I land the test run changes. There should probably be an “at long last” in that last sentence, but I’m just happy it’s done.
What are “test runs,” you might ask? Test runs allow the men and women behind the Mozilla QA curtain to direct the community testing effort a little more effectively than we could previously. We do this by creating what we call a “test run,” which in the context of Litmus amounts to a collection of testgroups that need running, a time frame within which those testgroups need to be run (i.e. a deadline), and a set of configurations that need testing.
e.g. As we approach a release for, say, Thunderbird 2, QA will receive a group of release candidate (RC) builds from the build team via tinderbox, one for each tier 1 platform (Win32, Mac, and Linux). At the very least, we’ll want to run a set of basic functional tests (BFTs) for each build on each platform. Using Litmus, the QA team can setup a test run that captures the build IDs and platform information of those builds, set a testing deadline, and then open that testing up to the Mozilla community at large. The test run report shows the QA team at a glance which configurations and testgroups/subgroups are being tested and which ones need more attention.
This mimics the process flow that the Mozilla QA team follows internally for release testing, and as I mentioned above, allows us to provide testers with more testing direction to help us out, especially near release times when we need that help the most.
In terms of the new interface, you’ll notice some changes immediately. Hopefully the first is speed. We have replaced the old start page that showed recent individual test results with a statistical roll-up for 5 active test runs. In my local testing, this has reduced the initial pageload time from >10s to around 3s. This was accomplished primarily by pushing the coverage look up (which was killing us before) into AJAX so that it loads after the page itself has loaded. Clicking on the coverage percentage will bring up the test run report.
The testing work flow is largely similar to what it was before, with the exception of the entry point.
Testers start out by selecting a test run. Test runs marked with a star 
are recommended by Mozilla, i.e. please start your testing here. If there is nothing recommended, or you simply want to test free-form as you always did, select the catch-all test run for the product and branch (1.5, 2.0, trunk) that you want to test. Click on the test run name to select that run for testing.
After selecting a test run, select from the available criteria and/or input your configuration information as appropriate. You will then be asked to select a testgroup and subgroup to test, and the interface from here on will be familiar to anyone who has tested with Litmus before.
There are already some issues that I’m aware of, e.g. reports for catch-all test runs don’t display accurate info., but I figure it’s better to roll this out and iterate on fixes rather than push this off even longer in search of perfection that won’t come. As such, feedback is welcome. Come find me in IRC (#qa or #litmus), or file bugs if you’re comfortable doing so.
