Tests are code, doctests aren't

In my last post I explained why I think narrative-style tests make poor unit tests. That alone is a good reason not to write unit tests in Python's doctest format. Here are more reasons why I don't like doctest for writing tests.

But that's not all. A more fundamental reason why I dislike doctests is that tests are code, and code works better in a .py file than a .txt file. There are a couple of reasons for this:

Tools can be improved to cope with doctest (for instance I heard that my pdb problems may be solved in Python 2.5), but new tools are continually being invented, and I want to be able to use those too. For instance, the 2to3 tool for converting Python 2.6 code to the upcoming Python 3.0 doesn't fix code in doctest files. And I still can't do “set filetype=doctest” in vim, which is hardly a new tool.

With sufficiently improved tool support and infrastructure many (but not all) of my concerns would be reduced. For instance, it would help if there were a way to easily reset all state during a long doctest, so that different parts of the same file could be independent. And then it would be good if there were also then a convenient way to put names on these independent sections. But you'd still be left with a design that gently encourages people to do things a worse way (write a big story), and you'd be reinventing the wheel: xUnit already gives you those things.

In my experience many developers with the best of intentions will produce poor unit tests with doctest because of the way it subtly encourages bad practices. One bad habit I've seen over and over again is copying-and-pasting helper functions, even large, complicated ones, from doctest to doctest. Is it because it's not “real” code, so the instinct to organise it and avoid duplication doesn't trigger? Is it because there's no obvious home for helper functions, because a doctest is not a module? I wish I knew.

I do not think doctests are evil. The doctest format is fine for some things. For “page tests” (e.g. using zope.testbrowser, as demonstrated here), where there's a narrative of a user story driving them, doctests are a pretty good fit. They can be good for writing testable documentation (which is not the same as tests and documentation mixed together!) too. But those things aren't unit tests.

I've mentioned this book a couple of times, and I do recommend it:

xUnit Test Patterns: Refactoring Test Code
Gerard Meszaros

You can find it on Amazon here.

If nothing else, reading it encourages thinking about the way you write tests, and ways you could do it better.

So despite the hype, I don't think doctest has an advantage over xUnit in producing readable tests. Code needs to be clear (including an appropriate amount of docstrings and comments) whether or not it's test code. If your developers aren't writing clear code, you have a serious problem: you are sure to have difficulty maintaining that code. It is just as possible to write incomprehensible tests using doctest as it is using TestCase classes with test methods. I know this because, unfortunately, I've seen plenty of both. Writing good tests is a skill that takes time and practice to learn. Using doctest is obviously not a silver bullet. Not using doctest isn't a silver bullet either, but I do think it's usually the better choice.

Andrew Bennetts, October 2008