mr mr cote

Educational, Investigative, and Absurd Writings by M. R. Côté

Bugzfeed: Bugzilla Push Notifications

| Comments

A large number of external applications have grown up around Bugzilla serving a variety of purposes. One thing many of these apps have in common is a need to get updates from Bugzilla. Unfortunately, the only way to get notifications of changes was, until recently, to poll Bugzilla. Everyone knows that polling is bad, particularly because it doesn’t scale well, but until recently there was no alternative.

Thus I would like to introduce to the world Bugzfeed, a WebSocket app that allows you to subscribe to one or more bugs and get pushed notifications when they change. It’s rather a small app, based on Tornado, and has a very simple interface, so it should scale quite nicely. It relies on a few moving parts to work, but I’ll start with the basics and explain the whole system later.

The production version is at ws://bugzfeed.mozilla.org. I also made a very simple (and ugly) example app for you to use and examine. A development version of Bugzfeed is available at ws://bugzfeed-dev.allizom.org; it’s tied to the development Bugzilla server, so it’s a good place to experiment if you’re a Mozilla contributor; you can make whatever changes you need to bugzilla-dev without worrying about messing with production data. You’ll need to get someone in #bmo on irc.mozilla.org to reset your password, since we periodically refresh and sanitize the database on bugzilla-dev, and email is disabled so you can’t reset it yourself.

(This makes me think that there should probably be a Bugzfeed instance tied to Landfill; maybe I’ll look into that, in particular if we implement middleware other than Pulse (see below).)

Client commands, responses, and notifications are all in JSON format. The project wiki page has the full list of commands. Here’s a little example of what you need to send to subscribe to bugs 1234 and 5678:

{"command": "subscribe", "bugs": [1234, 5678]}

The server will send a simple response, including a list of all the bugs you are (now) subscribed to:

{"command": "subscribe", "result": "ok", "bugs": [1234, 5678]}

Now you can just wait for notifications to be pushed from the server to your app:

{"command": "update", "bug": 1234, "when": "2014-04-03T21:13:45"}

Wait, you are probably asking, that’s it? That’s all I get?

The short answer is yup, that’s it. You can now use the regular REST API to get further details about what changed.

The longer answer is yup, that’s it, because security. Bugzilla has evolved a very fine-grained security system. We have bugs, attachments, and even comments that can only be seen by a privileged few, due to security, legal, and other considerations. Furthermore, many of the variables involved in determining whether a particular user can see a particular bug/attachment/comment can change at any time: not only can elements of a bug shift between public and confidential, but so can a user’s groups, and the groups themselves. Monitoring for all those possible changes would make this app significantly more complex and brittle, so we opted for the most secure notification, which is also the simplest: just a bug ID and a timestamp. All the other work is handled by the standard Bugzilla APIs.

(You might also be asking “why is ‘update’ considered a command?” and, to be honest, I’m not sure, so maybe that’ll change.)

There are other commands, and some limited caching of changes in case your client disconnects; see the project wiki page for more.


So how does it work? Here’s a system diagram created by contributor musingmario:

Bugzfeed system diagram

The four main pieces (with links to source) are

On the Bugzilla side, the BMO team created an extension which writes the bug ID and timestamp to a table when any bug changes. A simple Python app polls this table and sends all the updates to Pulse, cleaning up the table as it does so.

Pulse is a Mozilla RabbitMQ server with a specific configuration and message format implementing the publish/subscribe pattern. The usage is somewhat Mozilla specific, but it would be pretty easy to set up a similar system or even modify Bugzfeed and the Bugzilla shim to use RabbitMQ directly, or a different AMQP system like ØMQ.

Notifications from all bugs flow through Pulse; it is Bugzfeed, the WebSocket server, that does the filtering for its clients to notify only on subscribed bugs. Subscribing to individual notifications from Pulse is possible via topics, but this requires one channel per bug, so I doubt it would be any more efficient if hundreds of clients are connected to Bugzfeed.

While you could have the Bugzfeed server read directly from the Bugzilla database, eliminating the shim and the queuing system, having an intermediary allows us to easily stand up more Bugzfeed servers if load gets too high, as each Bugzfeed instance would see the stream of changes via its own subscriber queue. We can also easily interface new applications to the notification stream, such as the BMO Elastic Search cluster.

Enough technicalities; go out and play with it! And if you want to adapt it for your own Bugzilla installation, I’d be more than willing to help out.

Moving Bugzilla From Bazaar to Git

| Comments

Or, how to migrate to git using only three programming languages

Another aspect of Bugzilla has been dragged, kicking & screaming, into the future! On March 11, 2014, the Bugzilla source moved to git.mozilla.org. We’re still mirroring to bzr.mozilla.org (more on that later), but the repository of record is now git, meaning it is the only place we accept new code.

Getting over there was no small feat, so I want to record the adventure in the hopes that it can benefit someone else, and so I can look back some day and wonder why I put myself through these things.

Background

The rationale isn’t the focus of this post, but suffice it to say that Bazaar isn’t very widely used, and many projects are abandoning it. Eric S. Raymond wrote a good post on the Emacs dev list about why they need to move from Bazaar to git. The same rationale applies to Bugzilla: “Sticking to a moribund version-control system will compound and exacerbate the project’s difficulty in attracting new talent.”

So, moving on, I started off scouring the Internet to find the best way to perform this migration. One major complication is the fact that we want to keep mirroring (one-way) to Bazaar for at least a while, since the main suggested way to upgrade Bugzilla is from bzr.mozilla.org. It was deemed unreasonable to require existing installations to switch to git to obtain a small security fix, so we’ll continue to mirror changes to Bazaar for some time.

Initial migration

I found a few posts here and there about people who had done migrations like this, but the most useful was a post by David Roth from last year that detailed how to preserve Bazaar’s commit metadata, specifically bug-tracker metadata, which Bugzilla has used on virtually every commit since switching from CVS. It involves using the --no-plain option with bzr fast-export and then translating the output to something git understands.

Interestingly, Roth’s translation script was written in C#, not my personal first choice for such a task (or any, really, since I don’t generally develop for Windows). However it compiled fine under Mono, so I could run it on a Linux box. Something I learned, though, is to not try this kind of thing on OS X, where, by default, the filesystem is case-insensitive.

As much as I’d prefer to deal with a language with which I am more comfortable, I dislike duplicated effort even more. I used Roth’s C# script as a basis, modifying it a bit for our needs. The metadata is in the form <bug URL> <resolution>. Rather than editing existing commit messages, I just took that string and pasted it to the bottom of the commit message, but only if the bug number was not already in the commit message. This actually revealed a few typos in the “Bug 123456” strings that generally start commit messages.

There turned out to a few other subtle bugs, like the fact that a file which is both renamed and modified in the same commit shows up, in the output from bzr fast-export, as being modified under the original name. Thus if the delete is processed first, it looks like bzr has modified an nonexistent file. Those were easy to see by comparing the contents of every file before and after migration (admittedly just for the last revision).

Since there are a lot of branches on bzr.mozilla.org, I created a bash script to record them all and make sure none were missed. It output the pre-/postmigration diff of md5 sums as well as doing a git repack for each repo, after all branches were migrated.

One thing I forgot was pushing tags via the --tags option to git push; I had to do that manually after the migration. That’s also when I discovered that the same tag existed in several related bzr branches which were all combined into one git repo. This is, of course, not allowed in git. It made me think more about how Bugzilla uses certain tags, like current-stable, which are moved after each release. In git this requires the --force option to git push and is a big no-no if the remote repo is shared. I learned that, in fact, this is also the case in bzr, though perhaps it’s regarded as less of a sin than it is in git. Anyway, I’ve since realized that those should be branches, named appropriately (per branch). Despite them not being branches in the standard sense—they’ll always point to somewhere at or behind a version branch and never fork—it’s perfectly acceptable to move them, as opposed to tags, and since they’ll always be fast-forwarded, they won’t take any more space than a lightweight tag.

Mirroring

This was a harder problem. Originally, I tried to use the bzr-git extension, and it failed when I tried to pull in changes from git. I exchanged some emails with bzr-git’s author, Jelmer Vernooij, and he said that to keep an existing bzr branch in sync with a newly formed git repo is impossible at the moment: “This is essentially the ‘roundtripping’ support that we’ve been trying to achieve with bzr-git for a while. It’s a nontrivial problem (since it requires all semantics from bzr to be preserved when pushing to git).” Considering bzr-git hasn’t had a new release in two years, I won’t be holding my breath.

Luckily (and perhaps somewhat unfortunately) Bugzilla has jumped VCSes before, as I hinted above. With the old bzr-to-cvs script as a starting point, I created a git-to-bzr script—in, of course, Perl, as the original.

This script is essentially an automated way of applying individual commits from a git branch to a bzr branch. For each commit, the entire file tree is copied from a local git clone to a local bzr checkout, bzr add and remove are executed where needed, and the changes committed with the original author, contributor, and date preserved. The script also parses out the standard “Bug X:” commit-message format and passes it to bzr’s --fixes commit option. A file called .gitrev in the bzr repo tracks the associated git commit ID for each bzr commit.

To avoid excessive disk activity, since the script polls git and bzr for changes, the script uses bzr cat to check the contents of the .gitrev file and git ls-remote to get the ID of the last git commit. If they are equal, no further actions are performed.

Summing up

And that, folks, is how you can migrate from bzr to git! The initial migration is pretty straightforward, more so if you don’t care about any bzr commit metadata. It was unfortunate that there was no off-the-shelf way to sync the repos afterwards, but the basic idea isn’t too complicated.

For more, there’s the project page on our wiki, and all the scripts used are in a GitHub repo for your perusal. I’m no VCS expert—I’ve never heavily used bzr, and I’m constantly learning new things about git—but feel free to ask me questions if you want our process further clarified.

Vegan Bajan Tomato Sauce

| Comments

This isn’t a traditional recipe or anything, just something I put together one day. When we visit Barbados, we usually find a place with a kitchen, for various reasons: I like cooking, eating out every night is expensive (particularly in Barbados) and, now that I have a child with an early bedtime, getting out is a little tricky. But since we only stay for a week or two, buying all the spices I would normally use in tomato sauce is wasteful, and I never think to bring some with me. Hence one night my wife suggested using Bajan seasoning—a blend of shallots, garlic, hot peppers, and various herbs and spices. It’s a different taste, but quite delicious!

I use Delish Bajan seasoning, but I also found a recipe that I might try if I ever crave this recipe back home (although I will of course use a vegetarian Worcestershire sauce, such as The Wizard’s, athough it’s probably fine to leave it out, since I haven’t seen other recipes call for it). Bonus: that page has a recipe for hot sauce created by Barbados’s first Prime Minister, Errol Barrow!

The quantities below will serve about 3 people, but just double everything to serve 6.

  • olive oil
  • 1 small onion, chopped
  • 2-3 cloves of garlic, finely chopped
  • 5 fresh tomatoes, chopped according to preference (I like small dice)
  • 1 tbsp Bajan seasoning
  • 1 tsp salt
  • 1/3 package tofu

Optional:

  • 1 scotch bonnet pepper, finely chopped (or to taste—and I find the heat levels of fresh peppers can vary quite a bit here)
  • fresh or frozen spinach
  • black pepper
  • red wine
  • Bajan rum (for the chef, not the sauce)

Sauté the onion in the olive oil for a few minutes, until translucent. Add the garlic and sauté for another minute or so. Add tomatoes, Bajan seasoning, salt, and any optional ingredients. Mix well, bring to low boil, then reduce to a simmer. Cover pot and let cook for at least 15 minutes (longer is better, which gives a softer, velvety taste).

Meanwhile, steam the tofu. An easy way to do this is to put it in a bowl, add a teaspoon of water, cover, and microwave for 2-4 minutes (depending on your microwave). This dries out the tofu a little bit. You can also freeze and thaw tofu in advance for a similar effect. Fry it in some olive oil for 5 minutes or so, until it gets a bit chewy. I find (from vague recollection) that it can approximate the texture of ground beef. Add to sauce and mix well.

Serve over pasta of your choice.

BMO in 2013

| Comments

2013 was a pretty big year for BMO! I covered a bit in my last post on BMO, but I want to sum up just some of the things that the team accomplished in 2013 as well as to give you a preview of a few things to come.

We push updates to BMO generally on a weekly basis. The changelog for each push is posted to glob’s blog and linked to from Twitter (@globau) and from BMO’s discussion forum, mozilla.tools.bmo (available via mailing list, Google Group, and USENET).

I’m leaving comments open, but if you have something to discuss, please bring it to mozilla.tools.bmo.

Stats for 2013

BMO Usage:

35 190 new users registered
130 385 new bugs filed
107 884 bugs resolved

BMO Development:

115 code pushes
1 202 new bugs filed
1 062 bugs resolved

Native REST API

2013 saw a big investment in making Bugzilla a platform, not just a tool, for feature and defect tracking (and the other myriad things people use it for!). We completed a native RESTish API to complement the antiquated XMLRPC, JSONRPC, and JSONP interfaces. More importantly, we’ve built out this API to support more and more functionality, such as logging in with tokens, adding and updating flags, and querying the permissions layer.

Something worth taking note of is the bzAPI compatibility layer, which will be deployed in early Q1 of 2014. bzAPI is a nice application which implements a REST interface to Bugzilla through a combination of the legacy APIs, CSV renderings, and screen-scraped HTML. It is, however, essentially a proxy service to Bugzilla, so it has both limited functionality and poorer performance than a native API. With the new bzAPI compatibility layer, site admins will just have to change a URL to take advantage of the faster built-in REST API.

We are also planning to take the best ideas from the old APIs, bzAPI, the newly added functionality, and modern REST interfaces to produce an awesome version 2.0.

Project Kick-off Form

The Project Kick-off Form that was conceived and driven by Michael Coates was launched in January. The BMO team implemented the whole thing in the preceding months and did various improvements over the course of 2013.

The Form is now in the very capable hands of Winnie Aoieong. Winnie did a Project Kick-Off Refresher Brown Bag last month if you want, well, a refresher. We’ll be doing more to support this tool in 2014.

Sandstone Skin

BMO finally got a new default look this year. This was the result of some ideas from the “Bugzilla pretty” contest, the Mozilla Sandstone style guide, and our own research and intuition. BMO is still a far cry from a slick Web 2.x (or are we at 3.0 yet?) site, but it’s a small step towards it.

Oh and we have Gravatar support now!

User Profiles

Want to get some quick stats about a Bugzilla user—how long they’ve been using Bugzilla, the length of their review queue, or the areas in which they’ve been active? Click on a user’s name and select “Profile”, or go directly to your user profile page and enter a name or email into the search field.

File bugs under bugzilla.mozilla.org :: Extensions: UserProfile if there are other stats you think might be useful.

Review Suggestions and Reminders

Code reviews were a big topic at Mozilla in 2013. The BMO team implemented a couple related features:

  • Reviewer suggestions: When you are flagging a patch for review, you are now presented with a link to a list of one or more suggested reviewers according to the bug’s product and component. This is useful for new contributors who won’t necessarily know who would make a good candidate for review. Given beside the username is the number of reviews in that person’s queue, to encourage spreading reviews out.

  • Review notifications: As a result of a discussion on code reviews on dev.planning, by default you now get daily emails about your open reviews. You can also subscribe to these notifications for any Bugzilla user, something particularly useful to managers. As a bonus feature, you also get the number of requests assigned to you presented in a small red circle at the top of every Bugzilla page.

System Upgrade

When we upgraded† BMO to Bugzilla 4.2, IT also moved BMO from older hardware in Phoenix to new, faster hardware in SCL3. BMO was then set up anew in Phoenix and is now the failover location in case of an outage in SCL3.

† The BMO team regularly backports particularly useful patches from later upstream Bugzilla versions and trunk, but we fully upgraded to version 4.2 in the spring of 2013.

Other Stuff

We added user and product dashboards, implemented comment tagging, improved bug update times, and added redirects for GitHub pull-request reviews.

And then there were various bits of internal plumbing largely (by design!) invisible to users, such as the great tracking-flags migration; tonnes of little fixes here and there; and of course daily administration.

Plans for 2014

We’re already at work planning and implementing new features to start 2014 off right.

  • The Bugzilla Change Notification System will be deployed to production. This will allow external applications (and eventually the native UI) to subscribe to one or more bugs via Web Sockets and be notified when they change.

  • Performance instrumentation will be integrated into BMO (and upstream Bugzilla) to provide profiling data. Bugzilla’s been around for quite some time and, in supporting various complex workflows, its operations in turn can be quite involved. We’ll use data provided by this system to determine where we should focus optimization work.

  • We added memcached support to Bugzilla in Q4 of 2013; this will be pushed to BMO early in Q1 of 2014. Initially BMO will only use memcached for a few objects, but we’ll be adding more over time.

  • We’re setting up ElasticSearch clusters to provide a different way to access Bugzilla data, suitable for dashboards and general research.

  • Code reviews are a continued focus at Mozilla, so we’re implementing a way to get authoritative, comprehensive review data directly from BMO.

Our quarterly goals and other major work items are tracked on the BMO wiki page. You can also check out our road map for some vague ideas of plans into the future; these are ideas based on our current understanding of the Mozillaverse and will almost certainly change to some degree.

VMware Tools in Ubuntu

| Comments

I went about the seemingly simple task of sharing a directory in OS X with an Ubuntu VMware box so that I could code in my main desktop and run under Linux. The simple sharing dialog is of course only the beginning of the work; after that, I needed to refresh VMware tools, since I had done several kernel upgrades. Well that turned into a few hours of flailing at a command line.

For whatever reason, the kernel headers aren’t automatically found by the VMware Tools installation program, and even when you give the direct path, it still denies that they exist. Some web trawling told me that it looks for version.h, which isn’t in the root header directory, so I made a symlink. Then the installer found it but got compiler warnings a short time later.

I’m writing this post to tell you not to bother with any of that. Installing the open-vm-tools package from the multiverse repository is actually all you need! It’s amazing how many different searches it took for me to finally figure that out. I had to unshare and the reshare my directory for it to finally work, but now it’s working great. Let’s see what happens the next time I upgrade my kernel though…

ReviewBoard

There’s been a lot of interest in improving Mozilla’s code-review process lately, so in that vein the BMO team has set up a ReviewBoard instance at https://reviewboard.allizom.org for testing and evaluation.

ReviewBoard is a lot more useful than Splinter, so I suggest you try it out. One of the features I think will be most adored is proper interdiff support, made possible by the fact that ReviewBoard knows about the repo you’re working in. Tightly related is the ability to extend the context of the patch from the repo. Check out the ReviewBoard site for more.

Review apps, like most tools, are fairly contentious, so we wanted to give Mozillians a chance to try it out before we commit to it. Other options, like Phabricator, have been suggested; we happened to have been working on ReviewBoard. I’d like to hear from the greater public before settling on one for at least a few years.

For this experimentation phase, we’ve only done minimal integration with Bugzilla, namely, having ReviewBoard use Bugzilla’s accounts. You log into ReviewBoard with your Bugzilla account, and ReviewBoard’s username autosuggest is linked to Bugzilla (similarly, reviewboard-dev uses bugzilla-dev’s user database). (Note that this version of ReviewBoard doesn’t support Persona, but it should be there soon.) There’s a lot more we could do; some examples are in bug 515210. Again I’d like to hear feedback in order to prioritize our work.

I suggest using ReviewBoard much like we use GitHub pull requests. Start a review, then paste the URL as an attachment on a Bugzilla bug. Bug 922226 is on file to get redirects working for ReviewBoard reviews the way they do for pull requests.

For now, please don’t use ReviewBoard for any non-public (e.g. security-related) or really critical reviews. While the security team has gone over ReviewBoard, we’re still considering this an evaluation phase. We’ll also have to put some work into ensuring that only the right people can see non-public reviews; Bugzilla’s security system is rather fine-grained and complicated, so this will take some thought and possibly some modifications to ReviewBoard itself (don’t worry, we have several ReviewBoard developers in house!).

Finally, to get your repo added—having a linked repo is where you really see the value of ReviewBoard—either file a bug or drop by #bmo on IRC.

Please direct all feedback to mozilla.tools.bmo. To reinforce that, I’ve disabled comments on this post.

Mid-August BMO News

| Comments

A lot of people probably don’t know that I manage the team behind BMO, that is, bugzilla.mozilla.org, Mozilla’s Bugzilla installation. Work on BMO is continuous and incremental, and even really useful features often take a while to percolate through the community, so I thought I’d try to draw attention to some recent improvements that should get you pumped to open a Bugzilla tab.

Suggested Reviewers

A really exciting, and long-awaited, feature is suggested reviewers. Now when you flag a patch for review, you see a list of people who should make good candidates. And as of today, you can even see the number of reviews in each person’s queue! There is a lot more in this feature, with even more to come, so I’ll just run down a list:

  • The set of reviewers is per Product and per Component. If there are reviewers set for the particular Component, they are used; otherwise, it defaults to that Product’s reviewer set.
  • The full set is available as a report within Bugzilla. Anyone with “editcomponents” permission on a Product can change reviewers for that Product or its Components. Changes can also be requested by filing a bug or dropping by #bteam and talking directly to glob or dkl.
  • The order of the suggested reviewers is identical to how it appears in the report, which can be modified as above.
  • Anyone with “away”, “PTO”, or “unavailable” in their “real name” field will not be listed.
  • Any bug that has a mentor set ([mentor=X] in the whiteboard) will automatically have that user included in the suggested reviewers, at the top, in bold, since presumably that person is best suited for this particular bug.

Coming down the pipe is the ability to assign reviewers to subcategories of a particular component, which will show up beside the reviewer’s name in the list. Thus for components like Mochitest, you could indicate that a reviewer is particularly suited to b2g-specific parts of the framework. This avoids having to create tonnes of smaller components just for reviewing purposes.

Gravatars and the Sandstone Skin

There is now an option to display Gravatars beside comment headers. As with any Gravatar-enabled site, the avatar is displayed if the commenter’s bugmail address has an associated Gravatar. At the moment it is opt-in, enabled from the Preferences page (General Preferences / Show gravatar images when viewing bugs), but when the Sandstone skin becomes the default, Gravatars will be enabled for all users—but of course you can opt out if you really don’t like them.

Speaking of the Sandstone skin, some more improvements to it have landed recently. It’s pretty slick, in my opinion, and will become the default skin in a week or two when a couple remaining issues are resolved. For now, you can try it out by setting the General Preferences / Bugzilla’s general appearance (skin) option to Mozilla. Of course, after it is made the default, you can switch back to the old default (Dusk) if you prefer.

We have some more awesome changes landing in the next few weeks; I’ll post another summary then.

Autophone, a Case Study in Automating That Which Does Not Want to Be Automated (Part 1)

| Comments

Autophone is an automated system that executes Python test scripts on real user hardware, that is, actual phones. It’s been an active project for about a year now, and we’ve learned a lot about the difficulties of performing automated performance measurements on hardware that was never intended for automation. I’m documenting this story for posterity, since it has been an interesting, if often frustrating, experience. If you want to follow along, the source is on github.

I’m going to divide this into a few posts to hopefully avoid tl;driness and to ensure I actually finish at least some of this.

Background

For correctness testing, Mozilla’s Fennec automation is largely done with development boards, specifically Tegras and Pandas. These boards have wired power and ethernet, perfect for rack mounting, and all of a given type generally behave the same.

These boards are not, of course, consumer devices, and, despite having similar chipsets as real phones, they have different performance characteristics. To really see how Fennec performs in the real world, we need to measure Fennec on devices that people are buying and using every day. Thus was born the Autophone project.

Devices

At the moment, the existing Autophone production system runs tests on kind-of-to-very “old” phones. This isn’t a limitation of Autophone; rather, it’s a sampling of phones that were still very common last year, when the project got truly underway. We will add newer phones as time progresses, especially now that the system is very stable. The current system has at least one of the following phones:

  • HTC Google Nexus One
  • LG Revolution
  • Motorola Droid Pro
  • Samsung Galaxy SII
  • Samsung Galaxy SIII
  • Samsung Google Nexus S

We have a few more phones waiting to be deployed in a second cluster.

Architecture

A brief discussion of Autophone’s design may help in understanding the problems in automation and performance measurements by providing some context.

Autophone consists of a main Python process with one worker process per phone. We used processes instead of threads to isolate certain failures from other workers. The main process has separate threads for its TCP command server and for a pulse listener.

The worker processes are each tied to a single device and are responsible for controlling that device. The devices all have the SUTAgent and Watcher installed, to which the processes talk via mozdevice. A worker is spawned when the main process receives a SUT registration message on its command port from an unknown device. Device info is cached in a JSON file, and workers are also launched upon startup for any known device.

The workers each listen to their own queue, which receives commands from the main process. Commands come from users, via the command server, or are triggered by a build notification from pulse. The workers also check for jobs every 10 seconds (see below).

Autophone also includes a simple build cache server. This server handles requests for builds from the workers, fetching them via ftp as necessary, ensuring that only one copy of a particular build is downloaded at the same time, and that recently used builds are kept around, subject to space restrictions. (This part is actually very common to our automation frameworks, so it really should be extracted and put into its own module. Even better would be to extend mozdownload to support Fennec and have the build cache server use that to fetch builds. But I digress.)

Tests

When a build notification is received from pulse, or when a user issues a “triggerjobs” command, for each device an entry containing the build URL and device name is inserted into a sqlite database. A generic “job” command is then issued to each worker. As mentioned above, the workers poll this database every 10 seconds. They also poll the database immediately after executing a command, so the “job” command serves merely to trigger a poll of the database. This mechanism allows for worker processes to be restarted, since even if a “job” command is missed, the job itself will be picked up from the database. Along a similar vein, if the whole system is shut down, the current test will be restarted, and any queued tests will remain.

When a job successfully completes, the associated entry for that device and build is deleted from the jobs database. The number of attempts for each job is also recorded, and a job is abandoned after too many attempts, in case there are unrecoverable problems with a particular build or build/device combination.

Tests themselves are Python classes specified in a ManifestDestiny manifest file. They are (for better or for worse) executed in the worker process, i.e., not as a (further) separate process. Test classes are derived from a base class, PhoneTest, and are pretty much free form, requiring only a runjob() method that takes a dict of build metadata and the worker subprocess object, which can be used to manipulate the device as necessary, in particular to attempt to recover a frozen/disappeared device (though this part should probably be split into its own object, since a test shouldn’t be messing with the worker process object). The PhoneTest base class also provides some convenience functions to push profiles, start fennec, and report status messages to the main process.

At the moment, we have a single active test, S1S2, which measures Fennec startup performance. We also have support for a few unit tests (crashtests, reftests, JS reftests, mochitests, and robocop), though these are currently disabled pending some stability fixes.

Next post I’ll discuss the goal of S1S2, its challenges, and our solutions.

Whisky Cents

| Comments

If you’ve the means to buy an Oban 14 or a Macallan 18 whenever a decanter frees itself, you’ll probably want to saunter on past this post. However, if you like whisky but aren’t crazy about spending a good portion of your income on it, let me take a few minutes to tell you about some of my favourite everyday whiskies. I’m not going to include links because most distillers’ sites are obnoxious in one way or another, and I don’t have a favourite whisky-reviews site, but a quick search should point you in the right direction.

I’m far, far away from being a connoisseur of any kind—I have had several people tell me that regular consumption of hot peppers have killed most of my taste buds—but I have a basic appreciation of whisky, enough to at least be able to say what I like and what I don’t. I imagine that most people would consider the following to be “good” whiskies, but of course your tastes might—actually probably—vary from mine. Anyway, all this to say that I’m not going to embarrass myself trying to point out notes of green apple or hibiscus or library book and just confine myself to a few broad descriptors.

I’ve quoted current prices from Quebec’s liquor stores to give you an idea. You’ll probably pay less elsewhere in Canada, a fair bit less in most of the U.S., and don’t even talk to me about Europe.

The Tyrconnell Single Malt Irish Whiskey

I only briefly paused at this bottle the other day, but then I recalled the wonders of technology and fired up my mobile Internet gateway to find a few recommendations for the Tyrconnell. Emboldened, I brought a bottle home.

At $43.50, this is more grand larceny than a simple steal. It’s a light whiskey, no peat at all, a fruity nose with a honey taste. I haven’t tried a lot of Irish whiskey, but this is by far the smoothest that I’ve had.

I was also really excited to hear that its maker, Cooley Distillery, was one of the only independent distillers of single-malt whiskies in Ireland, but alas it was purchased by Beam Inc. a couple years ago. Oh well, still tasty.

Bruichladdich Waves

This is drying up in my local SAQs, but it’s been a favourite for a couple years now. It’s single malt but not vintage, meaning that it’s a blend of batches produced in the same fashion from the same sources.

In my opinion, the Waves has the perfect mix of peat and fruit. It’s a mild smoke on top of smooth caramel. Velvety goodness. It’s $55 a bottle, and I prefer it to the few other, more expensive Bruichladdichs I’ve tried.

As seems to be my fate, I was delighted that Bruichladdich (“Progressive Hebridean Distillers”) was one of, I believe, only two independent distilleries in Islay, but it was sold to Rémy Cointreau last year.

Amrut Indian Single Malt Whisky

I’ve only bought one bottle of this, but again it was due to recommendations online. It is a bit spicy, but not really in a piquant way, more like cloves or cinnamon. Quite comparable to a decent Scotch. Alas, the details have fallen out of my memory, but at $56.25 it’s only a buck and a quarter more than the Waves, so I’ll get another bottle for sure if I can’t find more of the Laddee.

Rebasing Etiquette

| Comments

I bet that the moment most people decide they actually do like git is when they start using ‘rebase’ regularly. I definitely do not completely understand the git model, but rebase shows that there is some seriously cool stuff going on.

Anyway, I’ve come upon a rebasing dilemma. The reasons for not rebasing a public repo are clear, but pushing to a remote origin (e.g. github) is also a form of backup. My master branches are for collaboration, but my dev branches are essentially just to back up my home computer, and occasionally for feedback. I rebase dev branches regularly, to keep my commits together for eventual merging to master. I occasionally switch around or squash commits too, where it adds clarity to the history. So, somewhat shamefully, I find myself using ‘git push -f’ a lot on branches other than master.

I guess I could get a paid account and fork private dev repos, but branches seem a lot more convenient, and I don’t really want to hide anything from anyone, as embarrasing as some of my work in progress sometimes seems to me.

Maybe the moral of this story is “never pull from my dev branches”?