[Pdx-pm] Modern testing in perl

Chad Granum exodist7 at gmail.com
Thu Mar 4 19:10:33 PST 2010


Recently there have been a couple movements to 'modernize' parts of
perl. Two immediate examples are Moose which is a more modern OO
system (like perl6), the other is perl5i which Schwern is heading.
perl5i is intended to fix all kinds of gripes. Thus far I have not
seen any similar movement in the area of perl testing. After a
discussion which compared and contrasted perl's testing tools with
another set of test tools it occurred to me that there is probably
room for significant improvement. I have decided to try to fill this
void in modernization of perl testing.

I have started the fennec project (http://github.com/exodist/Fennec)
before the name scares you let it be known that the original name was
to be Test::Suite, and I have just obtained that namespace (it was
previously owned by someone else) so the name may change back.

I am requesting that anyone and everyone put there 2 cents in on what
is amazing, good, bad, ugly, or impossible with the current perl
testing tools. How they can be improved, etc. I also encourage anyone
interested to add issues, look for bugs, add feature requests, submit
patches, fork the repo, etc.

Here is a simple bulleted list of desired features, most of these have
a current (bu maybe in need of improvement) implementation.

 * Group tests into sets which can be run multiple times under
different scenarios
 * Every test file should create an object that is run
 * Test sets should be run in random order by default, as should cases
(scenarios) and even test files.
 * Writing tester function libraries (think Test::More,
Test::Exception, etc) should be very simplified
 * Test results should be reported to the tester in object form,
unlike Test::Builder which just outputs any results directly to TAP.
 * Ability to create result handlers for cases where you want to get
results directly instead of going straight to TAP.
  * TAP output plugin used by default
  * Database output plugin (record results to a simple database)
  * Test::Builder output plugin (If you really want to go through
Test::Builder, this is also the first output plugin for quickstart of
the project)
 * Ability to use multiple output plugins at once.
 * Ability to wrap existing Test::Builder plugins (like Test::More)
into Fennec tester libraries (this is already done for Test::More,
Test::Warn)
 * Ability to run just a specified case/set
 * More helpful output in some current testers (is_deeply for example)
 * Ability to define tests both in separate files, and inline with the
objects being tested
  * When not in testing mode these definitions should be ignored and
minimal overhead should occur as a result of their being present.
  * Inline tests are purely optional
  * Perhaps tests be defined after __END__?

Test result objects need to contain the following information:
 * Name,
 * Result (ok, not ok),
 * Case run under,
 * Set run under,
 * Line tester was called from,
 * File tester was called from,
 * todo (false or reason),
 * skip (false or reason),
 * diagnostics messages,

Defining tests and cases should be moose like:

case name => sub {};
case name => (
   code => sub {},
   partition => 'name',
   ..options..,
);
set name => sub {};

Tests should also be definable by creating subs prefixed with 'test' or 'case'
sub test_something {}
sub case_prepare_things {}

There will also be an init method that will be called just once prior
to running the cases.

The base principal for Fennec is that test will be grouped into sets,
each of these sets can be run against multiple cases. Essentially a
case is a method on a test object you define that creates a scenario,
once the scenario is ready all your test sets will be run under that
scenario, once the sets are all completed the next scenario will be
run and the sets will be run again. You can mark an entire case or set
as todo or skip. You also can also specify that sets should only run
under certain cases, or not under others. You can also group sets and
cases into a 'partition' so that only sets in that partition will be
run under the cases in the same partition.

Currently there are 2 types of plugins, output plugins which take
results and do stuff with them, and tester plugins with provide
functionality such as ok, like, is, diag, etc.

I appreciate any feedback anyone wants to provide!

-Chad Granum


More information about the Pdx-pm-list mailing list