SPUG: Fwd: redux: misunderstanding (or bug) with $Test::More::TODO

BenRifkah Bergsten-Buret mail.spammagnet at gmail.com
Mon Dec 29 17:04:48 PST 2008

Whoops.  Forgot to include the list.

-- Ben

---------- Forwarded message ----------
From: BenRifkah Bergsten-Buret <mail.spammagnet at gmail.com>
Date: Mon, Dec 29, 2008 at 5:01 PM
Subject: Re: SPUG: redux: misunderstanding (or bug) with $Test::More::TODO
To: "Michael R. Wolf" <MichaelRWolf at att.net>

On Mon, Dec 29, 2008 at 2:29 PM, Michael R. Wolf <MichaelRWolf at att.net> wrote:
> I cannot trigger the documented behavior of a TODO block within Test::More
> using ok().  To test this, I have created two files.  Contrary to my
> understanding of the documentation (perldoc Test::More), I cannot get the
> two files to behave differently, either with perl(1) or prove(1).  Here is a
> complete listing of the files, and the output.

The "prove" output you included below shows that the files DO behave
differently just not in the way you expect.  It sounds like you are
expecting something that isn't in the documentation.

> I get an "ok" on the second test, regardless of perl/prove or inside/outside
> a TODO block.  (I *did* notice that the *comment* was different, but if it
> doesn't trigger a different *response* that's testable, so what?)

This is all as per the documentation.  It sounds like you're expecting
a TODO test that passes to say "not ok" but that isn't what the
documentation says.  The comment is part of the response and it is
testable because it contains a predictable string when a test is run
through a TODO block.  This is precisely how Test::Harness (prove)
detects "unexpected success" for TODO tests.

> I get a 0 return code (indicating success) in all 4 runs (2 files * 2 runs
> (1 each for perl(1) and prove(1)).  I was expecting that the "success" in a
> TODO would "report it as an unexpected success".  No dice.

I'm not sure why you're expecting a different return code for when the
tests in a given script passes or fails.  The return code is reserved
to tell you whether there were problems running the script and not
whether tests failed.

> $ perl t_m_trivial.t; echo $?
> 1..2
> ok 1 - normal true value 1
> ok 2 - normal true value 2
> 0
> $ prove t_m_trivial.t; echo $?
> t_m_trivial....ok
> All tests successful.
> Files=1, Tests=2,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.02 cusr  0.00
> csys =  0.05 CPU)
> Result: PASS
> 0

Note that in there is no report on the status of TODO tests in the
above prove output.

> $ perl t_m_trivial_w_todo.t; echo $?
> 1..2
> ok 1 - normal true value 1
> ok 2 - normal true value 2 inside a TODO block # TODO understand
> $Test::More::TODO
> 0
> $ prove t_m_trivial_w_todo.t; echo $?
> t_m_trivial_w_todo....ok
> All tests successful.
> Test Summary Report
> -------------------
> t_m_trivial_w_todo (Wstat: 0 Tests: 2 Failed: 0)
>  TODO passed:   2
> Files=1, Tests=2,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.02 cusr  0.00
> csys =  0.04 CPU)
> Result: PASS
> 0

Note here that the test summary report contains the "TODO passed:  2"
line.  This is the "unexpected success" report that I described in my
response to your initial query.  This is hinted at but not described
in the documentation.  As I mentioned above the exit statuses included
in these examples are unnecessary and the fact that they all exit 0
shows that the scripts ran to completion.

-- Ben

More information about the spug-list mailing list