Presentation?

[Darren Duncan]

On Thu, 12 Dec 2002, Peter Scott wrote:
> I could talk about Test:: modules if necessary. But I'd sooner see someone
> else step up to the plate this month.

Actually, module testing is something that I could use a little more help
with.  There are some questions here, but a discussion might be in order.

I understand that providing a test suite with any modules released
to the public / CPAN (or even used internally) is generally a must so that
it is easier for potential users of a module to be confident that it is
working correctly.

Also, it helps the developer be confident that when
they change some internals they can know that the outward behaviour is
still the same and/or changed to the way they expect.  For many modules
one doesn't have to spend much time retesting after even small changes,
because a quick run of a test script will just say yay or nay.  However, I
find that to get this kind of robustness it is taking me a lot of time and
effort to make my test suites in the first place.

The easiest kinds of
modules to write tests for are those which are self-contained (sit on top
of bare perl and don't use other modules) and particularly which are of
the data structure or data conversion variety, which only take input
through their methods and only send output through their methods.  Those 4
of my modules that are this type have a full test suite at the moment, and
the test count for each is about 229, 64, 45, 90, or somewhere between
those ranges; each test suite took me several full days to write.

By
contrast, it is modules that serve as interfaces to something external,
such as files or networks or user interfaces or databases or other
programs or hardware, which are a lot more difficult to write tests for,
because how do you exhaustively test things where a large part of which is
out of your control.  For example, how would I test certain kinds of
errors that can happen with a file, as another of my modules (with an
incomplete test suite) is supposed to do?  It is easier to show that it
works when it should than that it catches errors when there are external
problems.

For a related question, if a module is supposed to interface to a file,
such as perhaps create one or write to one, where should this file go such
that it doesn't step on other things in the filesystem?  Or, how does one
simulate a failure to do a read or a write or an flock; it is easy enough
to simulate an open failure (such as asking for a wrong name) by contrast.
Also, is it normal for test suites to come with other files for them to
manipulate other than what is in the test.pl code itself?

I will note that my 10+ modules which are not in the 'easier' category
mentioned above have been released with a test suite that usually does
nothing more than test that the module compiles.  Besides that, I don't
know for sure that it works except that I have observed the applications
running on top of them to behave correctly.  But those don't use every
single option that the modules provide for their completeness.

For the most part, it is the presence of a complete test suite that makes
the difference for my modules being in status 'R'eleased (with) or
'b'eta/'a'lpha (without).  I would really like to get my modules into 'R',
but I don't want to be deceptive by just putting them there anyway.  I
also don't want to take a lot of time to make the tests.

These are some issues I struggle with.  It is easy to make module updates
and updates to their documentation by contrast, as often module methods
are smaller than the test code for them.  And trying to have proper tests
is really slowing me down when I want to put something out quickly, taking
a lot of my time and delaying results.

Perhaps a more general question is, of all the CPAN modules, about what
fraction have an exhaustive test suite and what fraction have an
incomplete suite.  Is it acceptible to have modules in status 'R' that are
not fully tested.  Are in any modules in status 'a' or 'b' that have
comprehensive test suites and pass all the tests?

Any feedback or suggestions? -- Darren Duncan