Thursday, June 26, 2008

Doctest tweaking

Based on a long discussion thread, I'm in the midst of hacking up the NumPy test framework to make the following changes:
  1. All doctests in NumPy will have numpy in their execution context as "np" (i.e., there will be an implicit "import numpy as np" statement for all doctests, which makes the sample code shorter).
  2. The "normalize whitespace" option is enabled for all doctests. This one didn't come up in the discussion, but I ran across a few existing examples that fail because of insignificant whitespace differences (like a space after a calculation result), and figured it would save a lot of pain on the part of docstring editors. Especially since the default failure message doesn't show that whitespace is causing the test to fail. (I think this should also avoid having to use the ugly <BLANKLINE> in docstrings, have to check).
  3. Commands that have certain stuff in them will have their output ignored (but they will still be executed). At present the list of triggers is: ['plt.', 'plot.', '#random']. This avoids failures caused by output differences that aren't relevant: plotting function/method calls that return objects which (for some reason) include an address in their repr, examples that generate inherently random results, etc.
I may also turn on the detailed diff output for doctests as well; in some of the more complicated outputs, it's not always easy to spot some tiny difference in output. Come to think of it, the unittest fixture failure output could benefit from this, too, for the same reason.

All these changes will get run by the numpy-discussion list before they're actually committed, of course.