In my last post I explained why I think narrative-style tests make poor unit tests. That alone is a good reason not to write unit tests in Python's doctest format. Here are more reasons why I don't like doctest for writing tests.
sorted(...)when comparing dictionaries to get a deterministic comparision. This detracts from readability. In xUnit, a simple, obvious, and clear
assertEqualwould just work. In doctests, if this fails:
>>> foo == bar Truethen you get a completely unhelpful error, but doctest leaves you with little choice if you have dynamic values that vary between test runs. Again, this Just Works in xUnit with
assertEqual. In general, xUnit custom assertions are more flexible and readable than doctest's output matching. As Guido said on python-dev in July:
This is an example of the problem with doctest -- it's easy to overspecify the tests. I don't think that whether the repr() of a Decimal uses single or double quotes should be considered a spec cast in stone by doctests.
<BLANKLINE>”) is ugly. The syntax for toggling various doctest features inline (“
#doctest: +IGNORE_EXCEPTION_DETAIL”) is worse. The language is outright buggy in places — the following doctest passes:
>>> print 'hello' ... print 'world' helloThis one passes too:
>>> assert True ... garbage >>> print 1 1Testing APIs like pyunit can and do have ugly corners and bugs too, but the scope for problems is larger with a mini-language. I've never heard of an outright syntax error being silently ignored by pyunit! I might be more forgiving of doctest's quirks if it wasn't almost 10 years old already.
But that's not all. A more fundamental reason why I dislike doctests is
that tests are code, and code works better in a
.txt file. There are a couple of reasons for
.pyfiles correctly. Pdb works better with normal code (in doctests the capturing of stdout confuses the prompting). I can use standard profiling tools. I can run PyChecker and Pyflakes on
.pyfiles. I can use ctags. I can use bicyclerepairman. I can use pydoctor or epydoc. There are many more examples.
class Thing(object): """Docstring.""" # Comment.In doctests, you have to write
>>> class Thing(object): ... """Docstring.""" ... # Comment.Those tedious “
...” mean that almost every single code snippet I've seen in a doctest has lacked even a single comment or docstring, even when they really needed it. A prose preamble isn't always the best place to explain code.
Tools can be improved to cope with doctest (for instance I heard that my pdb
problems may be solved in Python 2.5), but new tools are continually being
invented, and I want to be able to use those too. For instance, the 2to3
tool for converting Python 2.6 code to the upcoming Python 3.0 doesn't fix code
in doctest files. And I still can't do “
set filetype=doctest” in
vim, which is hardly a new tool.
With sufficiently improved tool support and infrastructure many (but not all) of my concerns would be reduced. For instance, it would help if there were a way to easily reset all state during a long doctest, so that different parts of the same file could be independent. And then it would be good if there were also then a convenient way to put names on these independent sections. But you'd still be left with a design that gently encourages people to do things a worse way (write a big story), and you'd be reinventing the wheel: xUnit already gives you those things.
In my experience many developers with the best of intentions will produce poor unit tests with doctest because of the way it subtly encourages bad practices. One bad habit I've seen over and over again is copying-and-pasting helper functions, even large, complicated ones, from doctest to doctest. Is it because it's not “real” code, so the instinct to organise it and avoid duplication doesn't trigger? Is it because there's no obvious home for helper functions, because a doctest is not a module? I wish I knew.
I do not think doctests are evil. The doctest format is fine for some things. For “page tests” (e.g. using zope.testbrowser, as demonstrated here), where there's a narrative of a user story driving them, doctests are a pretty good fit. They can be good for writing testable documentation (which is not the same as tests and documentation mixed together!) too. But those things aren't unit tests.
I've mentioned this book a couple of times, and I do recommend it:
You can find it on Amazon here.
If nothing else, reading it encourages thinking about the way you write tests, and ways you could do it better.
So despite the hype, I don't think doctest has an advantage over xUnit in
producing readable tests. Code needs to be clear (including an appropriate
amount of docstrings and comments) whether or not it's test code. If your
developers aren't writing clear code, you have a serious problem: you are sure
to have difficulty maintaining that code. It is just as possible to write
incomprehensible tests using doctest as it is using
classes with test methods. I know this because, unfortunately, I've seen
plenty of both. Writing good tests is a skill that takes time and practice to
learn. Using doctest is obviously not a silver bullet. Not using doctest
isn't a silver bullet either, but I do think it's usually the better choice.
—, October 2008