Autophone is an automated system that executes Python test scripts on real user hardware, that is, actual phones. It’s been an active project for about a year now, and we’ve learned a lot about the difficulties of performing automated performance measurements on hardware that was never intended for automation. I’m documenting this story for posterity, since it has been an interesting, if often frustrating, experience. If you want to follow along, the source is on github.
I’m going to divide this into a few posts to hopefully avoid tl;driness and to ensure I actually finish at least some of this.
Background
For correctness testing, Mozilla’s Fennec automation is largely done with development boards, specifically Tegras and Pandas. These boards have wired power and ethernet, perfect for rack mounting, and all of a given type generally behave the same.
These boards are not, of course, consumer devices, and, despite having similar chipsets as real phones, they have different performance characteristics. To really see how Fennec performs in the real world, we need to measure Fennec on devices that people are buying and using every day. Thus was born the Autophone project.
Devices
At the moment, the existing Autophone production system runs tests on kind-of-to-very “old” phones. This isn’t a limitation of Autophone; rather, it’s a sampling of phones that were still very common last year, when the project got truly underway. We will add newer phones as time progresses, especially now that the system is very stable. The current system has at least one of the following phones:
- HTC Google Nexus One
- LG Revolution
- Motorola Droid Pro
- Samsung Galaxy SII
- Samsung Galaxy SIII
- Samsung Google Nexus S
We have a few more phones waiting to be deployed in a second cluster.
Architecture
A brief discussion of Autophone’s design may help in understanding the problems in automation and performance measurements by providing some context.
Autophone consists of a main Python process with one worker process per phone. We used processes instead of threads to isolate certain failures from other workers. The main process has separate threads for its TCP command server and for a pulse listener.
The worker processes are each tied to a single device and are responsible for controlling that device. The devices all have the SUTAgent and Watcher installed, to which the processes talk via mozdevice. A worker is spawned when the main process receives a SUT registration message on its command port from an unknown device. Device info is cached in a JSON file, and workers are also launched upon startup for any known device.
The workers each listen to their own queue, which receives commands from the main process. Commands come from users, via the command server, or are triggered by a build notification from pulse. The workers also check for jobs every 10 seconds (see below).
Autophone also includes a simple build cache server. This server handles requests for builds from the workers, fetching them via ftp as necessary, ensuring that only one copy of a particular build is downloaded at the same time, and that recently used builds are kept around, subject to space restrictions. (This part is actually very common to our automation frameworks, so it really should be extracted and put into its own module. Even better would be to extend mozdownload to support Fennec and have the build cache server use that to fetch builds. But I digress.)
Tests
When a build notification is received from pulse, or when a user issues a “triggerjobs” command, for each device an entry containing the build URL and device name is inserted into a sqlite database. A generic “job” command is then issued to each worker. As mentioned above, the workers poll this database every 10 seconds. They also poll the database immediately after executing a command, so the “job” command serves merely to trigger a poll of the database. This mechanism allows for worker processes to be restarted, since even if a “job” command is missed, the job itself will be picked up from the database. Along a similar vein, if the whole system is shut down, the current test will be restarted, and any queued tests will remain.
When a job successfully completes, the associated entry for that device and build is deleted from the jobs database. The number of attempts for each job is also recorded, and a job is abandoned after too many attempts, in case there are unrecoverable problems with a particular build or build/device combination.
Tests themselves are Python classes specified in a ManifestDestiny manifest file. They are (for better or for worse) executed in the worker process, i.e., not as a (further) separate process. Test classes are derived from a base class, PhoneTest, and are pretty much free form, requiring only a runjob() method that takes a dict of build metadata and the worker subprocess object, which can be used to manipulate the device as necessary, in particular to attempt to recover a frozen/disappeared device (though this part should probably be split into its own object, since a test shouldn’t be messing with the worker process object). The PhoneTest base class also provides some convenience functions to push profiles, start fennec, and report status messages to the main process.
At the moment, we have a single active test, S1S2, which measures Fennec startup performance. We also have support for a few unit tests (crashtests, reftests, JS reftests, mochitests, and robocop), though these are currently disabled pending some stability fixes.
Next post I’ll discuss the goal of S1S2, its challenges, and our solutions.