Coop » Mozilla

Setting descriptions to avoid buildbot exceptions from doStepIf

Coop — Tue, 20 Jul 2010 14:38:29 +0000

A small caveat (and workaround) if you’re using doStepIf with buildbot:

The Mozilla RelEng team recently upgraded to buildbot 0.8.0. This has allowed me to start using conditional buildbot steps by specifying the doStepIf parameter and a small helper function to check whether given properties have been set. I love it, and it opens up a whole new ways of working with our builds.

Unfortunately, the doStepIf implementation is new-ish and still broken in some ways. If steps are skipped, the finished() code is called without start() ever having been run, leaving you with potentially uninitialized variables. In my case, this manifests if I don’t specify a description for my conditional build step. Buildbot attempts to build a description for me by walking the properties, some of which may not be set if doStepIf is false.

The workaround is, of course, to always provide a description for any conditional build steps (easy), or wrap everything in a try block.

Reclaiming space on stage.mozilla.org – space reclaimed!

Coop — Mon, 19 Jul 2010 16:26:22 +0000

I posted this followup to the newsgroups a few weeks back, but things got kinda hectic with the end of quarter and the Summit so I never circled back here.

The cleanup crontabs for all products are in place on stage.mozilla.org now. We did see a little bit of bustage with the initial set of MAR expiry commands being improperly limited, but luckily KaiRo didn’t lose anything irreplaceable.

We didn’t reclaim as much space as originally estimated because we preserved the contrib dirs largely intact. We did still manage to recover almost 350G of space across all products, bringing our %used number on stage down to 79%. It was at 95% the day before we put these crontabs in place.

I’ve documented the new policy in the wiki: ReleaseEngineering:StageCleanupPolicy

We’ll still have to grow the disk eventually, or shuffle stuff around, but at least we have *something* in place policy-wise now.

Thanks to everyone who weighed-in on this longstanding issue.

How to make your own updates

Coop — Fri, 21 May 2010 19:15:13 +0000

Another positive outcome from the recent work to generate nightly updates on the buildslaves is that we now have update generation tools that work on Linux, Mac, and Windows. There been some interest in the past from other software projects about using our update tools, but I know that getting all of the mozilla source setup to build just these tools is non-trivial, especially if your software isn’t Mozilla-based to begin with.

I’ve packaged up the update tools for each platform from a recent mozilla-central nightly and put them under the xulrunner directory on ftp:

I’ve also updated our wiki documentation about how we make our nightly updates for Firefox to give people a fighting chance of figuring out how to use these tools for their own purposes.

Enjoy!

Reclaiming space on stage.mozilla.org

Coop — Thu, 13 May 2010 02:30:07 +0000

For those who want to skip to my specific proposals — there are 6 — for reclaiming space on stage.mozilla.org, please skip ahead to “Redux”, but if you’re going to comment, please read the whole thing.

Everyday we produce up to 17G worth of new nightly builds for Firefox across all branches. This includes all opt+debug builds in 75+ locales, each on 4 operating systems (OSes), each on 9 different project branches. We do reclaim much of this 17G as we retire l10n builds older than 1 week, but we are still creating 1.3G of new nightly en-US builds that we need to store (essentially) indefinitely. nthomas did some cleanup recently as part of bug 562261 and that has bought us some more time, but this inexorable increase will eventually overrun our disk capacity on the staging server. If we add to this disk usage by nightlies from other products which may have worse nightly hygiene habits and the expected increase in space requirements every night as we add 4 new OSes (adding Linux 64bit, Windows 64bit, OSX10.6 64bit and Android), the problem is magnified.

To date, our solution to this problem has been to do periodic cleanups, usually under duress like bug 562261 (which can be error-prone), or to simply buy more disk. As Justin notes, while disk space may be “cheap”, it is not an infinite resource, either in terms of upfront or management cost. We need an actual policy to govern how long we keep nightlies. We can then use that policy as a baseline to frame further discussions about keeping things for longer periods in special cases.

The good news: there *is* space that can be reclaimed: There are two types of wins to be had here:

One-time space recovery by deleting or archiving material that is no longer useful, or that can be kept offline and spun up as required.
Codified policy changes for automatically expiring old content.

We can tackle #1 (one-time space recovery) for Firefox by:

moving no-longer-supported releases, i.e. anything prior to 3.0 (including firebird) to a true archive. This will free up about 125G of space, and will have the added benefit of not housing unsupported builds next to supported builds, making them a little more difficult for people to stumble upon.
deleting nightly builds older than a certain date. The Firefox nightly directories are conveniently broken down by year, so it makes it easy to see how much disk space we could reclaim by deleting old nightlies:

2004 16G
2005 53G
2006 191G
2007 129G
2008 177G
2009 236G
2010 216G (so far)

At the risk of making others’ arguments for them, both previous times we attempted to come up with a stage cleanup policy (2006 & 2008) the major concern raised was a need to keep builds in perpetuity to allow regression detection via binary search.

So I say this: I’d like to hear from developers who have actually had to perform binary searches of nightlies to let me know exactly how far back in time they have had to go. In the absence of any other data, I’m going to suggest we remove all nightlies prior to 2007 simply because that corresponds with the start of the hg era (March 2007).

Please note, that when I say “remove” in this context of this discussion, I am advocating for “delete” but understand that I may need to settle for “archive” in whatever form that makes sense to IT (slow disk/tape/???).

Other non-Firefox projects will need to make their own decisions as to how/when to archive older releases, but those projects could also reclaim a lot of space by deleting their older nightlies. Here are the aggregate disk usage number for nightly builds of Calendar (both Sunbird & Lightning), Camino, Mozilla Suite (not even built since 2007), and XulRunner, broken down by year:

2001 472M
2002 6.9G
2003 6.7G
2004 23.2G
2005 57.6G
2006 79G
2007 106G
2008 114G

Here are the nightly usage numbers for Thunderbird:

2003 108M
2004 17G
2005 47G
2006 87G
2007 88G
2008 60G
2009 73G
2010 159G (so far)

Again, there are big space recovery wins to be had here, depending on far back we want our accessible, online repository of nightly builds to be.

In terms of policy changes to curb accumulation, I propose implementing three reforms, the first two of which come out of nthomas’ work in bug 562261.

First, we’ll script the automatic expiry of mar files for nightly builds older than 1 month. Only the most recent complete and partial MARs are required, so this gives us a more-than-adequate buffer to detect and fix problems with the nightly update system. We have proven steps from bug 562261 that can be easily cron-ed to run weekly on weekends or other periods when the staging server is (relatively) idle. This can be done across all products.

Second, the RelEng team will start purging the contents of old candidates directories as part of our release process. For those who are unaware, the candidates directory lives under the nightly directory and holds all the various release files (builds, source, signatures, logs) until we green-light the release. Once the release is official, the important contents are sync-ed over to the releases/ subdir and the candidates dir becomes mostly redundant, modulo a few important logging artifacts. We’ll delete the builds for all but the two most recent candidate dirs, but will preserve the text files/logs that tell us important things like # of builds/changesets/build IDs.

Note that this won’t be an automated procedure: the release engineer responsible for the current release will need to go in and look at the candidates directories involved and make a judgment call as to what to delete. Sometimes the release procedure goes awry or we try something new, and it’s important to be able to keep those examples around until we’ve learned what we can from them.

Other projects that currently use a candidates directory for releases should also consider making this change.

Third, we should agree to revisit nightly storage on a yearly basis. Specifically, we should commit to taking the oldest year’s worth of nightly builds offline in January of each year, e.g. if we’re comfortable with a 3-year online repository of nightly builds, in January 2011 we would take the nightly builds for 2007 offline. As you can see from the YTD numbers for 2010, this is unlikely to actually keep up with increasing storage needs, but it is better than nothing.

Redux:

Here’s a brief summary of my proposals for reclaiming space on stage to make it easier for people to respond *AFTER* reading the above:

1) Move Firefox releases that are no longer supported (< 3.0, including firebird) to separate storage.

2) Remove Firefox nightlies prior to 2007, freeing 260G. These can be deleted if they're not going to be used, or archived if we think they might.

3) Remove nightlies for products other than Firefox prior to 2007, freeing 174G. Again, "remove" can mean either deletion or archiving.

4) Automate the deletion of nightly MAR files older than one month. Only the most recent MAR files are required. This would be done across all products.

5) Delete builds from older candidates directories after official release. This will reclaim up to 13G per build attempt per release. This will be a manual process.

6) For every new year going forward, remove the oldest remaining year of nightlies, e.g. for a 3-year history of nightly builds, remove nightly builds from 2007 in January 2011. This will be a manual process.

Feedback is appreciated, either in the newsgroups or in bug 342972.

Timely nightly updates for mozilla-central l10n

Coop — Tue, 30 Mar 2010 06:46:13 +0000

I landed a patch last week that represents the culmination of several months worth of effort by me, but that also brings to fruition the full promise of l10n nightly updates, work that was begun by Armen almost a year ago.

History and motivation

For almost as long as there has been a Firefox browser, there has been a nightly update channel whereby bleeding edge developers could keep their browser up-to-date with the latest code changes from the past day. For lay people, imagine the security update notices you receive every month or so, but happening on a daily basis. There are changes made every day in the Mozilla codebase — bug fixes, new features, interface changes — so it’s important for us to have a vector to get these changes into testers’ hands as quickly as possible. Since the people that use these “nightly” builds are often developers or involved community members, the likelihood of us getting feedback on any code changes introduced in these builds is relatively high.

To minimize the download size of each nightly update for our testers, we offer what we call a “partial” update, where we provide only the accumulated binary code differences between the new nightly build and the previous one. If testers are more than one nightly update behind, we offer them a “complete” update, which is essentially just an archive of the latest nightly build packaged in a format that can be understood by the browser’s updating framework.

The developers and community members who use these nightly builds and updates don’t necessarily live in the U.S. or speak English as their first language, however. Armen took the first huge step towards better serving these users last year when he added nightly update support for localized builds (we call these builds l10n for short) on our core development branch, mozilla-central. For the first time ever, non-English users could get the same bleeding edge code as English speakers, but presented in their own language.

Useful, but at what cost

Mozilla has many different active code branches, all of which produce nightly builds for testing. Having proven its utility with the mozilla-central nightly builds, we wanted to extend the same l10n nightly update functionality to our other major branches, specifically the 1.9.2 and 1.9.1 code lines. However, after turning on nightly updates for the 1.9.2 branch, we soon discovered a bottleneck in our update process.

Since we first began offering nightly updates, all of our nightly updates had been generated on a single linux machine (or more recently, a single Linux VM). This was fine when there was only one locale to worry about, English (en-US). The update for each nightly build took between 2-3 minutes to generate, so even with 4 code branches (mozilla-central, 1.9.2, 1.9.1, CVS trunk) and 3 platforms (Linux, Mac OS X, Windows), updates would only ever take about 30 minutes to generate, total, and they would rarely be blocked because builds never finish at exactly the same time. Adding localized builds into the mix threw everything into disarray.

Mozilla prides itself on it’s localization story, but the sheer number of available localizations (around 75 per code branch) was swamping our poor, little update generation VM. As a rough estimate:

2.5 minutes/update x 3 update platforms x 75 locales = 562.5 minutes = >9 hours to generate all the nightly updates for one code branch

When we turned on the l10n nightly updates for the 1.9.2 branch, we had to make a number of changes immediately.

First, we certainly couldn’t consider turning on updates for another branch or nightly update generation would simply never catch up with itself. Because it would take over 18 hours to generate all the required updates for just two branches, we prioritized the English updates (en-US) so as not to degrade the existing nightly update experience for the consumers of English builds.

At this point, nightly updates for some locales were not timely at all, e.g. the Windows nightly build for the Chinese Traditional, Taiwan (zh-TW) locale would wait in the queue over 12 hours every day before having it’s partial update generated. In the timezones where a zh-TW localized build would most likely be used, this meant the base code of the “updated” build was always out-of-date by an entire day.

Adding a new platform, or even a new locale, was painful in a multiplicative rather than an additive way. Also, if we ever needed to re-spin a set of nightlies (which implies a new round of nightly updates), it meant that nightly update generation wouldn’t catch up for over 24 hours.

Make it right

Under these conditions, the decision to parallelize update generation was easy, the only question was how. We did consider simply adding more Linux VMs to generate updates, but that would have involved writing code to manage the pending update queue across multiple machines, or at the very least introducing an dedicated update generation machine per platform. The more straightforward solution would be to use the massive parallelization already provided by our existing build slave pool.

The first hurdle in generating our nightly updates on the build slaves was ensuring that the MAR (Mozilla ARchive) and binary diff tools worked on platforms other than Linux (which they now do) and were built automatically for branches on which we wanted to provide nightly updates (which they are now). There were also a bunch of build system changes that needed to happen to get MAR files uploaded automatically from the build slaves. After that, it was testing, lots and *lots* of testing. Last week, we finally got out of the testing phase and were able to deploy the changes for mozilla-central.

The numbers

There are always trade-offs when making a change like this, but honestly this is a pretty big win. Here are some typical l10n build scenarios from last week compared to the previous week before the updates-on-slave changes were in place:

Platform	Last Build Finished (before) PDT	Last Build Finished (now) PDT	Added time/build (m)	Update available (before) PDT	Update available (now) PDT
Linux	03:44	04:22	1.5 – 5	18:05	04:18
Mac OS X	06:10	06:24	2.5 – 4.5	16:55	06:24
Windows	05:45	06:58	3 – 4	19:50	06:58

In aggregate, we’ve added a maximum of 5 minutes/locale (often much less) x ~75 of locales, or 375 minutes, to the total time required to build all locales for a given branch on a given platform. Given 8 build slaves per platform assigned to create l10n nightly repacks, this means that the final l10n nightly repack on any build slave will finish roughly 45 minutes later than it did before the change. The parallelization of the overall process more than compensates though, at least as far as nightly updates are concerned. For example, the zh-TW localized Windows build may finish at 07:00 PDT now instead of 0615 PDT, but it’s nightly update is also immediately available at 07:00 PDT. English (en-US) builds also have their updates generated on the slaves, but since their updates were prioritized before anyway, they continue to be timely.

The localized builds also need to build the binary diff tool (bsdiff), but we re-use the same checkout for the tools from one l10n build to the next, so we only ever have to incur that build cost once, in the absence of a clobber. Even when we do need to rebuild bsdiff, since we also re-use the existing configuration steps and make targets that have already been run earlier in the l10n build process, the maximum time we ever spend building the bsdiff tool is about 1 min.

Onward

Perhaps what’s most exciting about this change are the possibilities it opens up for us, and for others.

Nick is already busy extending this work to provide nightly updates for project branches. Project branches are a relatively new thing in the Mozilla world, and soon the developers working on them will be able to stay current code-wise just the same as if they were working on mozilla-central. KaiRo also tells me that he’s only a few days away from having nightly partial updates available for SeaMonkey for the first time ever!

On 1-on-1s

Coop — Fri, 12 Mar 2010 21:19:20 +0000

A few weeks ago, dria blogged about the format she had adopted for conducting 1-on-1 meetings with her manager. As a nascent manager myself, I had been struggling to come up with a better framework for my own meetings. I ran dria’s chosen format by Armen and he was supportive, so for the past two weeks we’ve been giving it a try.

The new meeting format is working pretty well for us so far. Here are some of the key things I like about it:

The framework makes it easy to decide when in the meeting to discuss which issues. The project-based agenda I used previously tended to encourage digression, but that’s kept to a minimum now. Certain meeting sections are pure reporting (e.g. accomplishments) and others are meant for discussion (e.g. blockers).
It puts an onus on both parties to be prepared. Armen sends his notes to me the night before so I always have a chance to prepare help/advice beforehand.
The video chat is great as a remotie. Armen has an office he can go in to in Toronto, but I have no such luxury. Having these meetings “face-to-face” (i.e. Skype) is a big win, at least for me. Non-verbal communication can help us move faster through the agenda at times. Conversely, it can also allow me to pick up on signs that something may be amiss. Both functions are invaluable.

Our success with the format has encouraged me to push the format back up the chain: John and I are going to try the format next week for our 1-on-1.

GrowReviewComment

Coop — Fri, 26 Feb 2010 22:29:30 +0000

This has been a pet peeve of mine for a while. Assuming you’re not editing the attachment as a comment, Bugzilla gives you a very small window in which to leave a comment when reviewing a patch: the textarea is 5 rows high and only 25 columns wide by default. I’ve found it hard to add coherent commentary in a box that small, and my efforts almost always come out poorly formatted because I just can’t tell how the text is wrapping.

I whipped up a dead-simple greasemonkey script today that increases the size of the comment box when you click on it, and then shrinks it back down again when you click elsewhere.

Download: GrowReviewComment

Kiss the Future goodbye

Coop — Fri, 12 Feb 2010 23:49:49 +0000

Let it never be said that we don’t listen.

In response to the comments on my previous entry about Tackling the Release Engineering:Future queue and a number of conversations I’ve had out-of-band over the past few months, the release engineering team has decided to do away with the Future component. We will be merging all of the bugs therein back into the regular Release Engineering component.

Aside from the simplicity inherent in having fewer components to worry about, there were three main reasons we chose to do this:

No other group in the company/community has an explicit “Future” component where they can hibernate bugs.
People outside the RelEng team find the Future component opaque, and despite reassurances to the contrary, they often just assume bugs in the Future component won’t be fixed.
Moving bugs between the regular Release Engineering component and the Future component makes it very hard to extract meaningful statistics about how our bugs change over time (new bug rate, fix rate, etc.). This makes it hard to change our processes based on data.

What does this mean for you? Nothing, hopefully. I’m planning to move all the bugs out of the Future component on the morning (EST) of Monday, February 15th. People in North America will be on holiday, so this should be a relatively low-traffic time to make this change. My apologies in advance for the RelEng mail bomb you’ll be getting on Tuesday morning. :/

We will still continue to triage bugs using our three bucket approach, only the location of the third bucket will have changed. Our three buckets of bugs correspond to the following:

Untriaged: – state: NEW, assigned to: nobody, priority: none (–)
Triaged, assigned – state: NEW (we’re not working on it yet) or ASSIGNED (we’re working on it), assigned to: a specific release engineer, priority: P1-3
Triaged, unassigned (was Future) – state: NEW, assigned to: nobody, priority: P3-5

In the interest of further simplification, the RelEng team will eschew the Severity field as we move forward, although we won’t change any information already contained in that field if it exists. Instead, we’ll start using the Priority field more diligently. All bugs that have been triaged will have a Priority associated with them from here on out.

For reference, the various Priority levels mean the following to us:

P1: blocker, all other work on hold until this is fixed
P2: being actively worked on
P3: acknowledged as an issue, but not being actively worked on
P4: (future) goals that are either blocked on other work, or that simply have not been started yet
P5: enhancement

There was some concern within the RelEng team that developers might assume that bugs moved out of the Future component were now being worked on. I have faith that people will correctly interpret the implications of their bug still being assigned to nobody@mozilla.org as an indication that, no, their bug is still on the shelf.

Please note that this does not change the triage work that still needs to happen on these once-and-Future bugs, as described in the previous post. We’ve already made some good headway on triage this week, though. The number of “Future” bugs is now down to 304 from a high of 343. Kudos to the people who made that happen. Let’s keep up that great work.

Tackling the Release Engineering:Future queue

Coop — Mon, 08 Feb 2010 21:22:27 +0000

The Release Engineering:Future bugzilla component alternately inspires feelings of sadness, loathing, and contempt…and that’s just within the RelEng team!

I’m certain most developers first response to having their bug moved to the Future queue is, “Oh, look, my bug has fallen down a well.” Historically speaking, that may not be far from the truth.

What goes in Future?

Why does the Future component make people get punchy? For a long time, the Future component has lacked a decent description, so developers don’t know what it means when their bugs are moved there. Many have started hording their gigawatts in anticipation.

Bugzilla currently has the following small description of the Release Engineering:Future component:

“For longer term projects that have been agreed should be done, but have no immediate plans to so. These are not be part of the regular recurring triage. Advanced planning and placeholder goals for next quarter also go here.”

Despite the grammatical errors, this description is mostly accurate, but what does it actually mean:

Triaged bugs with no immediate owners go here.
Sadly, most enhancement bugs end up here unless they will make the core release process better, more streamlined, etc. quickly.
Bugs that are blocked on longer term projects in other groups go here until there is something for RelEng to do.
Advanced planning and placeholder goals for next quarter (or later) also go here. That’s pretty self-explanatory.

Remember: this description is for the Future component ONLY. The RelEng team continues to pick up and work on bugs that need to get done on a daily basis.

I’m thinking about how to improve this description, and will get the description updated in Bugzilla once I have achieved some rough consensus. In the meantime, I’ve posted this description of the Future component in the wiki as an ongoing reference.

The (Ever-Increasing) Numbers

At the time of writing, the Release Engineering:Future component has 343 bugs in it. This number has grown steadily over the past year despite having more release engineers on staff, and having made a great many improvements to our release automation and our continuous integration infrastructure.

Our turnaround time for bugs in the Future component is also not stellar. At our urging, the Mozilla Metrics team recently started setting up a dashboard to give us various statistics on our Bugzilla usage. Bugs don’t get fixed in the Future queue, so it’s hard to make truly accurate assessments here, but there are a lot of aging bugs in there. According to the numbers, fully 50% of the bugs in the Future component are older than 1 month and more than 25% are older than 6 months. How this compares to other teams or areas, I can’t say, but it certainly makes me empathize with developers who feel their Future-ed bugs have gone walkabout.

If it makes you feel better in a sadistic way, the majority of bugs filed by RelEng team members go directly into the Future queue. We’re not overly happy about it either.

I have a way? Is that better than a plan?

Things are getting done, though. For example, over the past month, the number of non-Future, release engineering bugs that are actively being worked on has gone from a low water mark of 130 up to over 200 today. For comparison, over the same period, the total number of bugs in the Future queue has slowly crept up from a low of 302 to 323. Mozilla is growing along so many axes that sometimes it feels like keeping the increase in the number of Future bugs to a linear relationship rather than exponential one is an accomplishment in itself.

So how do we stop the increase and start wrestling the Future component back under control?

The first step is triage. Starting this Thursday, February 11th, the RelEng team is going to have bi-weekly triage meetings to specifically prune down the Future queue.

As part of the triage, we’re going to be touching every bug and updating the whiteboard field with searchable tags. Our goal here is to make it easy to find classes of bugs in the Future queue so that duplicates and overlap can be easily eliminated, and fixes can be batched as much as possible.

We’ll be keeping a list of the tags we’re using in the wiki in case you want to follow along.

There are some classes of bugs that we won’t be able to eliminate (e.g. future goals), but hopefully within a few months, we’ll have the Future queue back under control.

It won’t be a quick fix, but it’s one we’re committed to.

All about the RelEng sheriff, revisited

Coop — Thu, 04 Feb 2010 19:37:46 +0000

This is a follow-up to Ben’s blog post about the RelEng Sheriff back in October. His post described clearly what the RelEng Sheriff (and more generally, the RelEng team) can do to help developers.

Since we implemented the RelEng sheriff (or “buildduty” as it is more informally called) last spring, developers have been getting better about using the buildduty person as the first point of contact for RelEng issues. I’d like to implore people to continue doing so. There are a few exceptions, of course:

It is outside of regular work hours and/or the designated buildduty person is not available; or,
you are already working on a bug with a specific release engineer.

We recognize that different release engineers have different core competencies, and it may be tempting to go “straight to the source,” but please respect our process. The process is in place to limit (as much as possible) the interrupt-driven nature of release engineering work and to minimize the impact of those interrupts on ongoing projects. In many cases, the person with the expertise you need will still end up helping you (we’re all in the same IRC channels, after all), but the buildduty person is specifically committed to helping you, even if only as a thin scheduling proxy for another release engineer.

Ben’s post had a good list things that RelEng can help developers with. Based on actual requests over the past few months, here are a few other services we can offer:

clone a virtual machine for one-off debugging/testing: Sometimes patches work on the try server but (for whatever reason) don’t once checked in. Sometimes tests fail or are intermittently orange. If you’re seeing issues like these after landing a patch, or need access to a specific platform for short-term work, the RelEng team can clone one of our reference machines and give you access (under certain circumstances).
grab build/extra logs from a build machine (provided they exist). This is often useful if a try server build generates extra, non-standard output

In both the above cases, the best thing to do is talk to the person on buildduty and then file a bug.

Conspicuously missing from Ben’s post was a list of what RelEng can’t do for you. While we can usually point you in the right direction, there are some things that we explicitly cannot do:

help you debug issues with your own local builds. We’re not (necessarily) build config experts, so unless you’re running with exactly the same configuration as our build machines and don’t have any code patches applied, you’d be better served asking the talented folk in #developers or posting to the newsgroups (m.d.builds or m.d.a.firefox)
shepherd your patch through the landing process. Standard patch landing rules apply: *you* are expected to be around when you land patches. If builds/tests fail and you’re not around, your patch will likely be backed out.

Need help finding the RelEng sheriff? Here’s the RelEng sheriff schedule.