Making sure tests pass with PQM
Wouldn't it be nice if tests always passed on the trunk of your project?
(I spoke about this a little at my OSDC talk last year, but I recently remembered that I ought to blog about it to get a wider audience.)
The benefits are pretty obvious: it saves people time. Bleeding edge users/testers can always be confident that a checkout of your project will build and work. Developers can start new feature branches confident that nothing is seriously broken before they've even started. Test coverage hopefully is improving over time, so you can be fairly sure that the quality of the trunk is also improving. And so on.
So how hard is it? Many projects have a policy that requires developers to run tests before checking in, and if it turns out they forgot and broke the test suite, their commit will be reverted. Well, it'll be reverted as soon as someone notices. And hopefully no-one checked in while you were running tests, which might easily happen if you have slow test suite.
Here's what the Launchpad and Bazaar developers do: use PQM (Patch Queue Manager) to guard commits
to the trunk. No developer has direct write access to the trunk branch, only
the PQM instance. To merge a branch to trunk, developers submit a merge request
to PQM. We configure PQM to run “make check” before
committing a merge, and only if the tests pass (and the merge had no conflicts)
will the change be committed.
You could configure your revision control system to use a pre-commit hook, but that doesn't tend to work very well unless the checks are instantaneous. It's nice to finish your work, send the merge request, and then be able to disconnect your laptop from the network and hop on a train without interrupting the processing of your branch. PQM accepts requests via GPG-signed email, and also sends the results via email (although in principle it could just as easily use something like XML-RPC). PQM will simply queue requests if they arrive while it is busy with an existing request. It's an asynchronous workflow, which works well when the checks can take some time.
If you have a bunch of inter-dependent projects, you can even configure PQM to run a command to check all of them every time someone tries to commit to any one of them. We do this with Launchpad, so we know that upgrading a dependency of the main Launchpad project won't cause problems in Launchpad itself.
It's not a perfect process, and PQM isn't a perfect tool. PQM in some ways is still quite simple and has rough edges, but it's worked very well for us. Our tests always pass on trunk. Occasionally there's a crisis like tests that normally pass but intermittently fail, but these would still be a problem without PQM, assuming you really do care about making sure tests always pass on your trunk.
This process can be complementary with the post-commit testing that many projects already do with BuildBot or Tinderbox. You could have PQM run tests on one or two key platforms, and then after the commit is done let BuildBot run tests on every other platform the community has buildslaves for. For instance, you may have a buildslave that tries to run your latest code with the latest development version of Python: failures there are worth knowing about, but not necessarily a reason to stop a branch from landing.
There's already a Bazaar plugin for sending requests to PQM, but you could pretty easily write plugins or scripts for other systems too.