[Omaha.pm] Benchmark.pm (cont'd from August)

[Jay Hannah]

Benchmark.pm:
   http://search.cpan.org/~rgarcia/perl-5.10.0/lib/Benchmark.pm

Back in August I complained. Yesterday I got interesting & useful info 
on irc.perl.org #poe:



12:44 <@Leolo> Benchmark: timing 1000000 iterations of amore, less, more...
12:44 <@Leolo>      amore:  4 wallclock secs ( 2.57 usr +  0.00 sys =  
2.57 CPU) @ 389105.06/s (n=1000000)
12:44 <@Leolo>       less:  2 wallclock secs ( 2.52 usr +  0.00 sys =  
2.52 CPU) @ 396825.40/s (n=1000000)
12:44 <@Leolo>       more:  2 wallclock secs ( 2.58 usr +  0.00 sys =  
2.58 CPU) @ 387596.90/s (n=1000000)
12:47 <@jhannah> Am I the only one that gave up on Benchmark?   
http://mail.pm.org/pipermail/omaha-pm/2008-August/001855.html
12:54 <@dngor> jhannah: Yes, wall clock is useless for benchmarking.  
That's why I use a CPU-time cutoff and compare the iterations/cpu time.
12:55 <@jhannah> wouldn't an accurate wallclock be insanely trivial to list?
12:55 <@Leolo> wallclock is useless
12:55 <@dngor> Wall time isn't accurate in a pre-emptive OS.
12:55 <@dngor> You're at the whim of other processes.
12:55 <@Leolo> other processes will hog the CPU, throwing off the .... 
what dngor said
12:55 <@dngor> Now, if you nice -19 the benchmark, that would make it 
better. :)
12:57 <@dngor> It may also require a reboot.  Be sure your benchmark 
will eventually exit. :)
12:57 <@jhannah> I understand the point you're making. What I don't 
understand why Benchmark would label something "wallclock" when it has 
no bearing, whatsoever, on how much clock-on-the-wall-time elapsed. It 
should be removed or accurate...?
12:57  * jhannah adds an "is" in there somewhere
12:58 <@dngor> Oh, I see.  Well, there's Benchmark.pm overhead which I 
presume it tries to discount.
13:00 <@dngor> I admit I'm just guessing at this point.  The CPU/wall 
ratio implies that the program wasn't pre-empted much.
13:01 <@dngor> Program run time, from start to exit, also involves 
loading Perl and the source, compiling it, and tearing down the 
process.  Those may be significant, but Benchmark.pm won't report them.
13:02 <@dngor> Whether these apply, I don't know.
13:03  * jhannah hugs purl, shoots Benchmark
13:03 <@jhannah> call it anything EXCEPT "wallclock" and I'm happy.  :)
13:07 <@dngor> Example of a cpu-time limit: 
http://poe.dyndns.org/~troc/benchmarks/bench-calls.perl
13:08 <@dngor> Example of "rolling your own" for jhannah: 
http://poe.dyndns.org/~troc/benchmarks/arguments.perl
13:11 <@jhannah> dngor: so you just ignore the "wallclock secs"? Or you 
trust them as accurate relative to each other but unrelated to a clock 
on the wall?
13:11 <@dngor> I ignore them and look at n per X cpu seconds.
13:11 <@dngor> CPU time is the only objective measurement.
13:12 <@dngor> The CPU is the universe.
13:13 <@dngor> There is no coupling at all between the CPU and the wall 
clock.  Consider that many machines dynamically clock now to conserve power.