[Dahut-pm] [RFC] Test reporting framework

[James G Smith]

A few weeks ago, I received a new supervisor.  I haven't changed jobs
or positions, but my previous supervisor had too many direct reports,
so he hired someone to take care of the application developers.

After talking with this new person about my work (especially the
application framework that saw it's broken 0.01 release a couple
months ago), he suggested that I put together some reports that show
dependencies between parts of the project and how far along we are on
those parts.  This would help us demonstrate a need for more people
working on the project as well as show where they could work.

Being a lazy Perl programmer, I didn't want to write information down
twice, so I have developed a set of scripts that do a lot of the work
for me.

I'm thinking about putting a lot of this together into a package for
CPAN.  I'm looking for feedback in a number of areas: naming of
package, information in reports, anything else that might be useful
that I haven't thought about.  I'm toying with the idea of adding
benchmarking capabilities similar to the tests embedded in Pod.  We
could probably do some blame tracking (who last touched a method,
etc.) using CVS.


  make_spec.pl

This is a script that will go through any existing specification that
might have been created by this script before, any modules listed on
the command line, and the MANIFEST (looking for files beginning with
lib/).  It will then create the spec.xml file which lists all the
modules, their dependencies within the project, and their public
methods.

For example:

<specification>
  <module name="Gestinanna::Util">   
    <interface>
      <method name="path2regex"/>
      <method name="path_cmp">
        <dependence method="path2regex"/>
      </method>
    </interface>
  </module>
  <module name="Gestinanna::Authz">
    <interface>
      <constructor/>
      <method name="can_grant"/>  
      <method name="fetch_acls"/>
      <method name="grant"/>
      <method name="has_attribute"/>
      <method name="query_acls"/>
      <method name="query_attributes"/>
      <method name="query_point_attributes"/>
      <method name="set_point_attributes"/>
    </interface>
    <dependence module="Gestinanna::Util"/>
    <dependence module="Gestinanna::Schema"/>
  </module>
</specification>

This documents that I have to test path2regex before testing path_cmp
since path_cmp depends on path2regex.  The method dependencies have
to be entered by hand.  The module dependencies do not if they are
documented with a `use' statement in the code.  The default
constructor is `new'.  If a module has constructors, they are tested
first at the moment.


  make_tests.pl

This script will generate all the t/ tests based on the
specification, honoring the module dependencies.  A module is only
tested after its dependencies are tested.  The methods/functions
within a module are done likewise.  Test results are tallied on a
per-method/function basis even though all the functions in a module
are tested in one file (beauty of Test::Builder and Module::Build :).

Additional tests are created to test module compilation, pod
correctness and coverage, and to handle any cleanup required after
running the tests.

Tests are pulled from the specification file and the module Pod.  The
Pod is processed using Pod::Tests.  Documentation coverage is handled
by Pod::Cover.

I do have a few special symbols for tests embedded in Pod.
__OBJECT__ is the default object created by the constructor test.
__PACKAGE__ is the package being tested; the tests run in main::.


  make_crc.pl

This script generates a series of reports based on the test results.
An example can be seen at
http://gorgik.tamu.edu/gestinanna/crc/index.html .  We use Perl::Tidy
to handle the line counts.  Gestinanna::Util is completely tested and
documented.  Gestinanna::Authz demonstrates a module with
constructors, and both public and private methods being tested.
--
James Smith <JGSmith at TAMU.Edu>, 979-862-3725
Senior Software Applications Developer,
Texas A&M CIS Operating Systems Group, Unix