mr mr cote

Educational, Investigative, and Absurd Writings by M. R. Côté

Project Isolation

| Comments

The other day I read about another new Mozilla project that decided to go with GitHub issues instead of our Bugzilla installation (BMO). The author’s arguments make a lot of sense: GitHub issues are much simpler and faster, and if you keep your code in GitHub, you get tighter integration. The author notes that a downside is the inability to file security or confidential bugs, for which Bugzilla has a fine-grained permission system, and that he’d just put those (rare) issues on BMO.

The one downside he doesn’t mention is interdependencies with other Mozilla projects, e.g. the Depends On/Blocks fields. This is where Bugzilla gets into project, product, and perhaps even program management by allowing people to easily track dependency chains, which is invaluable in planning. Many people actually file bugs solely as trackers for a particular feature or project, hanging all the work items and bugs off of it, and sometimes that work crosses product boundaries. There are also a number of tracking flags and fields that managers use to prioritize work and decide which releases to target.

If I had to rebut my own point, I would argue that the projects that use GitHub issues are relatively isolated, and so dependency tracking is not particularly important. Why clutter up and slow down the UI with lots of features that I don’t need for my project? In particular, most of the tracking features are currently used only by, and thus designed for, the Firefox products (aside: this is one reason the new modal UI hides most of these fields by default if they have never been set).

This seems hard to refute, and I certainly wouldn’t want to force an admittedly complex tool on anyone who had much simpler needs. But something still wasn’t sitting right with me, and it took a while to figure out what it was. As usual, it was that a different question was going unasked, leading to unspoken assumptions: why do we have so many isolated projects, and what are we giving up by having such loose (or even no) integration amongst all our work?

Working on projects in isolation is comforting because you don’t have to think about all the other things going on in your organization—in other words, you don’t have to communicate with very many people. A lack of communication, however, leads to several problems:

  • low visibility: what is everyone working on?
  • redundancy: how many times are we solving the same problem?
  • barriers to coordination: how can we become greater than the sum of our parts by delivering inter-related features and products?

By working in isolation, we can’t leverage each other’s strengths and accomplishments. We waste effort and lose great opportunities to deliver amazing things. We know that places like Twitter use monorepos to get some of these benefits, like a single build/test/deploy toolchain and coordination of breaking changes. This is what facilitates architectures like microservices and SOAs. Even if we don’t want to go down those paths, there is still a clear benefit to program management by at least integrating the tracking and planning of all of our various endeavours and directions. We need better organization-wide coordination.

We’re already taking some steps in this direction, like moving Firefox and Cloud Services to one division. But there are many other teams that could benefit from better integration, many teams that are duplicating effort and missing out on chances to work together. It’s a huge effort, but maybe we need to form a team to define a strategy and process—a Strategic Integration Team perhaps?

Pulse Update

| Comments

After languishing for a few years, Pulse got a burst of interest and development in 2014. Since I first heard of it, I’ve found the idea of a central message bus for the goings-on in Mozilla’s various systems rather intruiging, and I’m excited to have been able to grow it over the last year.

Pulse falls into that class of problem that is a result of, to borrow from a past Mozilla leader, our tendency to make our lives difficult, that is, to work in the open. Using RabbitMQ as a generic event stream is nothing special; Mozilla’s use of it as an open system is, I believe, completely unique.

Adapting a system intended for private networks into a public service always results in fascinating problems. Pulse has a decent permission-control system, but it’s not designed for self service. It is also very trusting of its users, who can easily overwhelm the system by just subscribing to streams and never consuming the messages.

The solution to both these problems was to design a management application: PulseGuardian. Via Persona, it handles account management, and it comes with a service that monitors Pulse’s queues. Since we presume users are not malicious, it sends a friendly warning when it notices a queue growing too large, but if ignored it will eventually kill the queue to save the system.

If you build it, they will come, or so says some movie I’ve never seen, but in this case it appears to be true. TaskCluster has moved whole-hog over to Pulse for its messaging needs, and the devs wrote a really nice web app for inspecting live messages. MozReview is using it for code-review bots and autolanding commits. Autophone is exploring its use for providing Try support to non-BuildBot-based testing frameworks.

Another step for Pulse beyond the prototype phase is a proper library. The existing mozillapulse Python library works decently, aside from some annoying problems, but it suffers from a lack of extensibility, and, I’m beginning to believe, should be based directly on a lower-level amqp or RabbitMQ-specific Python package and not the strange, overly generic kombu messaging library, in part because of the apparent lack of confirm channels in kombu. We’re looking into taking ideas from TaskCluster’s Pulse usage in bug 1133602.

Recently I presented the State of Pulse to the A-Team. I should do that as a general brownbag at some point, but, until then, you can look at the slides.

BMO 2014 Statistics

| Comments

Everyone loves statistics! Right? Right? Hello?

tap tap

feedback screech

Well anyway, here are some numbers from BMO in 2014:

BMO Usage:

33 243 new users registered
45 628 users logged in
23 063 users performed an action
160 586 new bugs filed
138 127 bugs resolved
100 194 patches attached

BMO Development:

1 325 bugs filed
1 214 bugs resolved

Conclusion: there are a lot of dedicated Mozillians out there!

BMO 2014 Update Part II

| Comments

The second half of 2014 was spent finishing up some performance work and shifting into usability improvements, which will continue into 2015.

More performance!

By the end of 2014, we’d managed to pick most of the low-to-medium-hanging fruit in the world of Bugzilla server-side performance. The result is approximately doubling the performance of authenticated bug views. Here are graphs from January 2014 and October 2014:

The server now also minifies and concatenates JavaScript and CSS files. This affects cold loads mostly, since these files are cached, but even on reload it saves a few round trips.

As mentioned above, we’re shifting focus away from performance work and towards usability/work-flow improvements, but there will still be perf benefits, both by reducing and delaying loading of content and by making it easier for users to accomplish common tasks.

New, better documentation

We’ve converted the upstream Bugzilla documentation to reStructuredText, massively updated and reorganized it, and, perhaps of most interest to anyone reading this, completely rewrote the API docs, which were very hard to grok.

For BMO specifically, we’ve fixed up the wiki page. We’ve also started a user guide, but it’s just a skeleton at the moment. There are lots of users out there who know the ins and outs of BMO, so feel free to contribute a section!

GMail support

To support Mozilla’s transition to GMail, we added two features. First, we now limit the number of emails sent to a user per minute and per hour, since GMail will temporarily disable accounts that receive too much mail, and some BMO users receive a lot of bugmail. Second, since GMail’s ability to filter mail by headers is limited compared to other email servers, users can now include the X-Bugzilla-* headers in the body via General Preferences.

Other things

  • When entering a new bug, after selecting the product, any relevant custom bug-entry forms are now displayed at the top of the form. We also added a full list of all the custom forms in BMO (linked at the bottom of the bug-entry landing page).

  • There’s now a UI for your Review History (as reviewer).

  • BMO now uses the Fira Sans typeface to improve consistency with other Mozilla web properties, although we reverted to the old monospace style after a lot of negative feedback about Fira’s monospace. If you don’t like Fira, you can always switch to OpenSans by changing your skin in the General Preferences.

  • My Dashboard now has an “Interesting Bugs” query. You can see a full description of the bug criteria in the bug’s user story, but in effect it should be close to what you would receive in bugmail. This is useful on its own if you don’t want to rely on bugmail, but we have plans to make it even more useful by adding weighted scoring to order bugs according to how relevant the changes are to you in particular.

  • There’s now an easy way to generate a shortened link for search URLs, which tend to be long.

  • And just landed yesterday, which is technically 2015 but the result of work done in 2014, is a new look for the guided bug-entry form, aka the Bugzilla Helper, which new BMO users get by default when filing new bugs. It’s the first step of a series of improvements intended to make it easier for new users to file good bugs.


As I’ve said a few times now, in 2015 we’re going to do a lot of work on improving general BMO-user productivity: usability, UX, work flows, whatever you want to call it. I’ll write more about this later, but here are a few things we’re looking into:

  • Experimenting with alternative bug views. There’s a lot of stuff on the standard bug view. Can we give different windows into the data depending on what the user is trying to accomplish? Can we load this data more intelligently, and organize it more intuitively?

  • A proper field for MozReview links that dynamically loads and displays state information on the review request.

  • Authentication via GitHub. A lot of Mozilla projects are on GitHub, but many use Bugzilla for issue tracking. To make it easier for contributors coming to us via GitHub, we’ll let them log into Bugzilla using their GitHub account.

  • As I wrote about in an earlier post, improved searchability.

As usual, if you have questions or comments, you can leave them here, but an even better place is the mailing list, also available as a Google Group and via NNTP.

Searching Bugzilla

| Comments

BMO currently supports five—count ‘em, five—ways to search for bugs. Whenever you have five different ways to perform a similar function, you can be pretty sure the core problem is not well understood. Search has been rated, for good reason, one of the least compelling features of Bugzilla, so the BMO team want to dig in there and make some serious improvements.

At our Portland get-together a couple weeks ago, we talked about putting together a vision for BMO. It’s a tough problem, since BMO is used for so many different things. We did, however, manage to get some clarity around search. Gerv, who has been involved in the Bugzilla project for quite some time, neatly summarized the use cases. People search Bugzilla for only two reasons:

  • to find a set of bugs, or
  • to find a specific bug.

That’s it. The fact that BMO has five different searches, though, means either we didn’t know that, or we just couldn’t find a good way to do one, or the other, or both.

We’ve got the functionality of the first use case down pretty well, via Advanced Search: it helps you assemble a set of criteria of almost limitless specificity that will result in a list of bugs. It can be used to determine what bugs are blocking a particular release, what bugs a particular person has assigned to them, or what bugs in a particular Product have been fixed recently. Its interface is, admittedly, not great. Quick Search was developed as a different, text-based approach to Advanced Search; it can be quicker to use but definitely isn’t any more intuitive. Regardless, Advanced Search fulfills its role fairly well.

The second use of Search is how you’d answer the question, “what was that bug I was looking at a couple weeks ago?” You have some hazy recollection of a bug. You have a good idea of a few words in the summary, although you might be slightly off, and you might know the Product or the Assignee, but probably not much else. Advanced Search will give you a huge, useless result set, but you really just want one specific bug.

This kind of search isn’t easy; it needs some intelligence, like natural-language processing, in order to give useful results. Bugzilla’s solutions are the Instant and Simple searches, which eschew the standard Bugzilla::Search module that powers Advanced and Quick searches. Instead, they do full-text searches on the Summary field (and optionally in Comments as well, which is super slow). The results still aren’t very good, so BMO developers tried outsourcing the feature by adding a Google Search option. But despite Google being a great search engine for the web, it doesn’t know enough about BMO data to be much more useful, and it doesn’t know about new nor confidential bugs at all.

Since Bugzilla’s search engines were originally written, however, there have been many advances in the field, especially in FLOSS. This is another place where we need to bring Bugzilla into the modern world; MySQL full-text searches are just not good enough. In the upcoming year, we’re going to look into new approaches to search, such as running different databases in tandem to exploit their particular abilities. We plan to start with experiments using Elasticsearch, which, as the name implies, is very good at searching. By standing up an instance beside the main MySQL db and mirroring bug data over, we can refer specific-bug searches to it; even though we’ll then have to filter based on standard bug-visibility rules, we should have a net win in search times, especially when searching comments.

In sum, Mozilla developers, we understand your tribulations with Bugzilla search, and we’re on it. After all, we all have a reputation to maintain as the Godzilla of Search Engines!

Review Board Preview

| Comments

I know lots of people are very anxious to see Mozilla’s new code-review tool. It’s been a harrowing journey, but we are finally knocking out our last few blockers for initial deployment (see tracking bug 1021929). While we sort those out, here’s something to whet your palate: a walk through the new review work flow.

BMO Mid-2014 Update

| Comments

Here’s your mid-year report from the offices, basements, and caverns of BMO!


This year we’re spending a lot of time on performance. As nearly everyone knows, Bugzilla’s an old Perl app from the early days of the Web, written way before all the technologies, processes, and standards of today were even dreamt of. Furthermore, Bugzilla (including BMO) has a very flexible extension framework, which makes broad optimizations difficult, since extensions can modify data at many points during the loading and transforming of data. Finally, Bugzilla has evolved a very fine-grained security system, crucial to an open organization like Mozilla that still has to have a few secrets, at least temporarily (for security and legal reasons, largely). This means lots of security checks when loading or modifying a bug—and, tangentially, it makes the business logic behind the UI pretty complex under the hood.

That said, we’ve made some really good progress, starting with retrofitting Bugzilla to use memcached, and then instrumenting the database and templating code to give of reams of data to analyze. Glob has lots of details in his post on BMO perf work; read it if you’re interested in the challenges of optimizing a legacy web app. The tl;dr is that BMO is faster than last year; our best progress has been on the server side of show_bug (the standard Bug view), which, for authenticated users, is about 15% faster on average than last year, with far fewer spikes.

Bugs updated since last visit

Part of an effort to improve developer productivity, in June we rolled out a feature to give a new way for users to track changes to bugs. BMO now notes when you visit a bug you’re involved in (when you load it in the main Bugzilla UI or otherwise perform actions on it), and any changes to that bug which occur since you last visited it will show up in a table in My Dashboard. Read more.

Bugmail filtering

Another improvement to developer productivity centred around notifications is the new bugmail filtering feature. Bugzilla sends out quite a lot of mail, and the controls for deciding when you want to receive a notification of a change to a bug have been pretty coarse-grained. This new feature is extremely customizable, so you can get only the notifications you really care about.

BzAPI compatibility

There have been several broad posts about this recently, but it’s worth repeating. The original Bugzilla REST API, known as BzAPI, is deprecated in favor of the new native REST API (on BMO at least; it isn’t yet available in any released version of the Bugzilla product). If possible, sites currently using BzAPI should be modified to use the new API (they are largely, but not entirely, compatible), but at a minimum they should be updated to use the new BzAPI compatibility layer, which is hosted directly on BMO and sits atop the new REST API. The compatibility layer should act almost exactly the same as BzAPI (the exceptions being that a few extra fields are returned in a small number of calls). At some point in the not-too-distant future, we’ll be (transparently) redirecting all requests to BzAPI to this layer and shutting down the BzAPI server, so it’s better to try to migrate now while the original BzAPI is still around, in case there are any lingering bugs in the compatibility layer.

More stuff

As usual, you can see our current goals and high-priority items for the quarter on the BMO wiki page.

Bugzfeed: Bugzilla Push Notifications

| Comments

A large number of external applications have grown up around Bugzilla serving a variety of purposes. One thing many of these apps have in common is a need to get updates from Bugzilla. Unfortunately, the only way to get notifications of changes was, until recently, to poll Bugzilla. Everyone knows that polling is bad, particularly because it doesn’t scale well, but until recently there was no alternative.

Thus I would like to introduce to the world Bugzfeed, a WebSocket app that allows you to subscribe to one or more bugs and get pushed notifications when they change. It’s rather a small app, based on Tornado, and has a very simple interface, so it should scale quite nicely. It relies on a few moving parts to work, but I’ll start with the basics and explain the whole system later.

The production version is at ws:// I also made a very simple (and ugly) example app for you to use and examine. A development version of Bugzfeed is available at ws://; it’s tied to the development Bugzilla server, so it’s a good place to experiment if you’re a Mozilla contributor; you can make whatever changes you need to bugzilla-dev without worrying about messing with production data. You’ll need to get someone in #bmo on to reset your password, since we periodically refresh and sanitize the database on bugzilla-dev, and email is disabled so you can’t reset it yourself.

(This makes me think that there should probably be a Bugzfeed instance tied to Landfill; maybe I’ll look into that, in particular if we implement middleware other than Pulse (see below).)

Client commands, responses, and notifications are all in JSON format. The project wiki page has the full list of commands. Here’s a little example of what you need to send to subscribe to bugs 1234 and 5678:

{"command": "subscribe", "bugs": [1234, 5678]}

The server will send a simple response, including a list of all the bugs you are (now) subscribed to:

{"command": "subscribe", "result": "ok", "bugs": [1234, 5678]}

Now you can just wait for notifications to be pushed from the server to your app:

{"command": "update", "bug": 1234, "when": "2014-04-03T21:13:45"}

Wait, you are probably asking, that’s it? That’s all I get?

The short answer is yup, that’s it. You can now use the regular REST API to get further details about what changed.

The longer answer is yup, that’s it, because security. Bugzilla has evolved a very fine-grained security system. We have bugs, attachments, and even comments that can only be seen by a privileged few, due to security, legal, and other considerations. Furthermore, many of the variables involved in determining whether a particular user can see a particular bug/attachment/comment can change at any time: not only can elements of a bug shift between public and confidential, but so can a user’s groups, and the groups themselves. Monitoring for all those possible changes would make this app significantly more complex and brittle, so we opted for the most secure notification, which is also the simplest: just a bug ID and a timestamp. All the other work is handled by the standard Bugzilla APIs.

(You might also be asking “why is ‘update’ considered a command?” and, to be honest, I’m not sure, so maybe that’ll change.)

There are other commands, and some limited caching of changes in case your client disconnects; see the project wiki page for more.

So how does it work? Here’s a system diagram created by contributor musingmario:

Bugzfeed system diagram

The four main pieces (with links to source) are

On the Bugzilla side, the BMO team created an extension which writes the bug ID and timestamp to a table when any bug changes. A simple Python app polls this table and sends all the updates to Pulse, cleaning up the table as it does so.

Pulse is a Mozilla RabbitMQ server with a specific configuration and message format implementing the publish/subscribe pattern. The usage is somewhat Mozilla specific, but it would be pretty easy to set up a similar system or even modify Bugzfeed and the Bugzilla shim to use RabbitMQ directly, or a different AMQP system like ØMQ.

Notifications from all bugs flow through Pulse; it is Bugzfeed, the WebSocket server, that does the filtering for its clients to notify only on subscribed bugs. Subscribing to individual notifications from Pulse is possible via topics, but this requires one channel per bug, so I doubt it would be any more efficient if hundreds of clients are connected to Bugzfeed.

While you could have the Bugzfeed server read directly from the Bugzilla database, eliminating the shim and the queuing system, having an intermediary allows us to easily stand up more Bugzfeed servers if load gets too high, as each Bugzfeed instance would see the stream of changes via its own subscriber queue. We can also easily interface new applications to the notification stream, such as the BMO Elastic Search cluster.

Enough technicalities; go out and play with it! And if you want to adapt it for your own Bugzilla installation, I’d be more than willing to help out.

Moving Bugzilla From Bazaar to Git

| Comments

Or, how to migrate to git using only three programming languages

Another aspect of Bugzilla has been dragged, kicking & screaming, into the future! On March 11, 2014, the Bugzilla source moved to We’re still mirroring to (more on that later), but the repository of record is now git, meaning it is the only place we accept new code.

Getting over there was no small feat, so I want to record the adventure in the hopes that it can benefit someone else, and so I can look back some day and wonder why I put myself through these things.


The rationale isn’t the focus of this post, but suffice it to say that Bazaar isn’t very widely used, and many projects are abandoning it. Eric S. Raymond wrote a good post on the Emacs dev list about why they need to move from Bazaar to git. The same rationale applies to Bugzilla: “Sticking to a moribund version-control system will compound and exacerbate the project’s difficulty in attracting new talent.”

So, moving on, I started off scouring the Internet to find the best way to perform this migration. One major complication is the fact that we want to keep mirroring (one-way) to Bazaar for at least a while, since the main suggested way to upgrade Bugzilla is from It was deemed unreasonable to require existing installations to switch to git to obtain a small security fix, so we’ll continue to mirror changes to Bazaar for some time.

Initial migration

I found a few posts here and there about people who had done migrations like this, but the most useful was a post by David Roth from last year that detailed how to preserve Bazaar’s commit metadata, specifically bug-tracker metadata, which Bugzilla has used on virtually every commit since switching from CVS. It involves using the --no-plain option with bzr fast-export and then translating the output to something git understands.

Interestingly, Roth’s translation script was written in C#, not my personal first choice for such a task (or any, really, since I don’t generally develop for Windows). However it compiled fine under Mono, so I could run it on a Linux box. Something I learned, though, is to not try this kind of thing on OS X, where, by default, the filesystem is case-insensitive.

As much as I’d prefer to deal with a language with which I am more comfortable, I dislike duplicated effort even more. I used Roth’s C# script as a basis, modifying it a bit for our needs. The metadata is in the form <bug URL> <resolution>. Rather than editing existing commit messages, I just took that string and pasted it to the bottom of the commit message, but only if the bug number was not already in the commit message. This actually revealed a few typos in the “Bug 123456” strings that generally start commit messages.

There turned out to a few other subtle bugs, like the fact that a file which is both renamed and modified in the same commit shows up, in the output from bzr fast-export, as being modified under the original name. Thus if the delete is processed first, it looks like bzr has modified an nonexistent file. Those were easy to see by comparing the contents of every file before and after migration (admittedly just for the last revision).

Since there are a lot of branches on, I created a bash script to record them all and make sure none were missed. It output the pre-/postmigration diff of md5 sums as well as doing a git repack for each repo, after all branches were migrated.

One thing I forgot was pushing tags via the --tags option to git push; I had to do that manually after the migration. That’s also when I discovered that the same tag existed in several related bzr branches which were all combined into one git repo. This is, of course, not allowed in git. It made me think more about how Bugzilla uses certain tags, like current-stable, which are moved after each release. In git this requires the --force option to git push and is a big no-no if the remote repo is shared. I learned that, in fact, this is also the case in bzr, though perhaps it’s regarded as less of a sin than it is in git. Anyway, I’ve since realized that those should be branches, named appropriately (per branch). Despite them not being branches in the standard sense—they’ll always point to somewhere at or behind a version branch and never fork—it’s perfectly acceptable to move them, as opposed to tags, and since they’ll always be fast-forwarded, they won’t take any more space than a lightweight tag.


This was a harder problem. Originally, I tried to use the bzr-git extension, and it failed when I tried to pull in changes from git. I exchanged some emails with bzr-git’s author, Jelmer Vernooij, and he said that to keep an existing bzr branch in sync with a newly formed git repo is impossible at the moment: “This is essentially the ‘roundtripping’ support that we’ve been trying to achieve with bzr-git for a while. It’s a nontrivial problem (since it requires all semantics from bzr to be preserved when pushing to git).” Considering bzr-git hasn’t had a new release in two years, I won’t be holding my breath.

Luckily (and perhaps somewhat unfortunately) Bugzilla has jumped VCSes before, as I hinted above. With the old bzr-to-cvs script as a starting point, I created a git-to-bzr script—in, of course, Perl, as the original.

This script is essentially an automated way of applying individual commits from a git branch to a bzr branch. For each commit, the entire file tree is copied from a local git clone to a local bzr checkout, bzr add and remove are executed where needed, and the changes committed with the original author, contributor, and date preserved. The script also parses out the standard “Bug X:” commit-message format and passes it to bzr’s --fixes commit option. A file called .gitrev in the bzr repo tracks the associated git commit ID for each bzr commit.

To avoid excessive disk activity, since the script polls git and bzr for changes, the script uses bzr cat to check the contents of the .gitrev file and git ls-remote to get the ID of the last git commit. If they are equal, no further actions are performed.

Summing up

And that, folks, is how you can migrate from bzr to git! The initial migration is pretty straightforward, more so if you don’t care about any bzr commit metadata. It was unfortunate that there was no off-the-shelf way to sync the repos afterwards, but the basic idea isn’t too complicated.

For more, there’s the project page on our wiki, and all the scripts used are in a GitHub repo for your perusal. I’m no VCS expert—I’ve never heavily used bzr, and I’m constantly learning new things about git—but feel free to ask me questions if you want our process further clarified.

Vegan Bajan Tomato Sauce

| Comments

This isn’t a traditional recipe or anything, just something I put together one day. When we visit Barbados, we usually find a place with a kitchen, for various reasons: I like cooking, eating out every night is expensive (particularly in Barbados) and, now that I have a child with an early bedtime, getting out is a little tricky. But since we only stay for a week or two, buying all the spices I would normally use in tomato sauce is wasteful, and I never think to bring some with me. Hence one night my wife suggested using Bajan seasoning—a blend of shallots, garlic, hot peppers, and various herbs and spices. It’s a different taste, but quite delicious!

I use Delish Bajan seasoning, but I also found a recipe that I might try if I ever crave this recipe back home (although I will of course use a vegetarian Worcestershire sauce, such as The Wizard’s, athough it’s probably fine to leave it out, since I haven’t seen other recipes call for it). Bonus: that page has a recipe for hot sauce created by Barbados’s first Prime Minister, Errol Barrow!

The quantities below will serve about 3 people, but just double everything to serve 6.

  • olive oil
  • 1 small onion, chopped
  • 2-3 cloves of garlic, finely chopped
  • 5 fresh tomatoes, chopped according to preference (I like small dice)
  • 1 tbsp Bajan seasoning
  • 1 tsp salt
  • 1/3 package tofu


  • 1 scotch bonnet pepper, finely chopped (or to taste—and I find the heat levels of fresh peppers can vary quite a bit here)
  • fresh or frozen spinach
  • black pepper
  • red wine
  • Bajan rum (for the chef, not the sauce)

Sauté the onion in the olive oil for a few minutes, until translucent. Add the garlic and sauté for another minute or so. Add tomatoes, Bajan seasoning, salt, and any optional ingredients. Mix well, bring to low boil, then reduce to a simmer. Cover pot and let cook for at least 15 minutes (longer is better, which gives a softer, velvety taste).

Meanwhile, steam the tofu. An easy way to do this is to put it in a bowl, add a teaspoon of water, cover, and microwave for 2-4 minutes (depending on your microwave). This dries out the tofu a little bit. You can also freeze and thaw tofu in advance for a similar effect. Fry it in some olive oil for 5 minutes or so, until it gets a bit chewy. I find (from vague recollection) that it can approximate the texture of ground beef. Add to sauce and mix well.

Serve over pasta of your choice.