NumPy doctest customizations committed

Since nobody objected to my warning post, the following features are now available for all NumPy 1.2 doctests:

1. All doctests in NumPy will have the numpy module available in their execution context as "np".
2. The normalized whitespace option is enabled for all doctests.
3. Adding "#random" to expected output will now cause that output to be ignored, but the associated example command will still be executed. So you can now do this to provide a descriptive example but not have it fail:
   

   >>> random.random()
   0.1234567890 #random: output may differ on your system
   

Later today I'll commit some changes to existing doctests that take advantage of these changes. Apparently some of the doctests haven't been run in a long time; tensorsolve, for example, referenced an undefined "wrap" function, and there were no unit tests to turn up this problem. :/ We're going to add a flag to the buildbot test command to run the doctests as well from now on, so that might help improve coverage a bit.

By the way, many thanks to whomever installed nose on the NumPy buildbots! :)

Oops, nevermind about <BLANKLINE>

One should not post first thing in the morning, it seems. Avoiding use of <BLANKLINE> in doctests has nothing to do with whitespace normalization. :)

Doctest tweaking

Based on a long discussion thread, I'm in the midst of hacking up the NumPy test framework to make the following changes:

1. All doctests in NumPy will have numpy in their execution context as "np" (i.e., there will be an implicit "import numpy as np" statement for all doctests, which makes the sample code shorter).
   
2. The "normalize whitespace" option is enabled for all doctests. This one didn't come up in the discussion, but I ran across a few existing examples that fail because of insignificant whitespace differences (like a space after a calculation result), and figured it would save a lot of pain on the part of docstring editors. Especially since the default failure message doesn't show that whitespace is causing the test to fail. (I think this should also avoid having to use the ugly <BLANKLINE> in docstrings, have to check).
   
3. Commands that have certain stuff in them will have their output ignored (but they will still be executed). At present the list of triggers is: ['plt.', 'plot.', '#random']. This avoids failures caused by output differences that aren't relevant: plotting function/method calls that return objects which (for some reason) include an address in their repr, examples that generate inherently random results, etc.
   

I may also turn on the detailed diff output for doctests as well; in some of the more complicated outputs, it's not always easy to spot some tiny difference in output. Come to think of it, the unittest fixture failure output could benefit from this, too, for the same reason.

All these changes will get run by the numpy-discussion list before they're actually committed, of course.

Old test stuff restored

All the old (1.1) test classes are back in NumPy. I can run old-style tests without any troubles locally, and hopefully that will hold for all the user test suites out there.

The module .test() functions will now also take the old arguments (which fortunately were passed in as keyword arguments in all the cases I've seen so far), and warn that they should be updated before the next release. As before, test() returns a TextTestResult; this required a little bit of tweaking on nose, because the test result object is discarded by default. Apparently nobody ever needed it back before, and that's why it's not kept. This might be changed in nose at some point in the future.

Unfortunately there's no way to tell if the new test suite works on the various buildbot machines, since none of them apparently have nose installed. :(

I managed to inadvertently add coverage support when I made my first big checkin. Playing around with experimental stuff in your working directory at the last minute FTL. :/ Anyway, Robert Kern was nice enough to point out some points on which it was lacking, and those are fixed. Running numpy.test(coverage=True) will now report coverage limited to NumPy, and numpy.core.test will limit coverage reporting to numpy.core, etc.

A quick test shows that it's not hard to change SciPy to use numpy.testing, allowing the removal of the code in scipy/testing.

NumPy has switched to nose

Ok, so I checked in my nose changes...and all the buildbots are red, and people's code is broken. :( So I have to do what I should have done in the first place and make sure tests written with the old test framework will still function in 1.2.

At least the switch turned up some useful information: since nose picks up tests that weren't being run by the original test suite, it uncovered some old code and tests in numpy/f2py/lib/parser which can probably be deleted. Well, maybe; still waiting on a definite answer on that. Deleting the two files (Fortran2003.py and test_Fortran2003.py) doesn't seem to cause any problems, and gets rid of some verbose test output.

NumPy nose migration

Ok, now that finals are over, the only disruptions in my day should be realtors showing potential buyers around, and the occasional packing spree.

I'm currently reviewing the changes to switch NumPy to nose before I check them in. Right now my local numpy.test() picks up and runs 1656 tests, with no failures or errors. Since I'm touching a lot of files, I figure a final scan over the diffs can't hurt. I already found a couple of tests that weren't being run under the old framework because they weren't defined correctly.

I had to add --exclude arguments to ignore the directories below numpy/numpy/distutils/tests (f2py_ext, etc.), since those little test extension modules don't get built by default. These tests weren't run by the old test suite anyway, but maybe it would be nice to see if they can be included later on.

I really like that nose has coverage support (many thanks to whoever added that). I wonder how hard it would be to implement some sort of coverage diff to let people check how a patch or local modifications to their NumPy tree will affect coverage.

Avoiding __test__ usage

I ran across some existing tests in NumPy today that made me realize there's no particular reason that classes like LinalgTestCase need to be derived from unittest.TestCase. Not sure why that didn't occur to me earlier. Anyway, if the classes with type-specific test methods don't derive from TestCase, nose won't try to run them. I think that this:

class LinalgTestCase:
    def test_single(self):
        a = array([[1.,2.], [3.,4.]], dtype=single)
        b = array([2., 1.], dtype=single)
        self.do(a, b)

class test_inv(LinalgTestCase, TestCase):
    def do(self, a, b):
        a_inv = linalg.inv(a)
        assert_almost_equal(dot(a, a_inv), identity(a.shape[0]))

is much cleaner-looking than the __test__ solution from June 2nd.