[Pdx-pm] Measuring memory consumption of scalars

[Marvin Humphrey]

David Wheeler writes...

>> Does this help?
>>
>>    http://search.cpan.org/dist/Devel-Size/Size.pm

Devel::Size has worked out well.

The only downside is that it looks like it fails 1 out of 13 smoke  
tests, so that failure will cascade onto Sort::External.

Eric Wilhelm writes...

> Looks like a good source of info and verification, but maybe not
> something that you want to put in your production code:
>
>   "Devel::Size, because of the way it works, can consume a
> considerable
> amount of memory as it runs. It will use five pointers, two integers,
> and two bytes worth of storage, plus potential alignment and bucket
> overhead, per thing it looks at."

The extra storage overhead turns out not to be all that much.

In the new version of Sort::External, sortex objects track of the  
size of the input cache by calling total_size(\@_) every time you feed 
() them.

I tried two different approaches with a c. 10_000-element source  
array (each item a 60-character string): feeding the items to the  
sortex object one by one, and feeding them all at once.  I set up two  
long Benchmark loops, with the one_by_one approach first.

Watching top, I saw the memory consumption of the process stabilize  
at 7528k during the one_by_one phase, then rise to a plateau of 8072k  
during the
all_at_once phase.  That's acceptable.

The performance penalty, as anticipated, diminishes in importance one  
tempfiles are required.  If you don't hit disk, and you're feeding  
items one-by-one, using total_size makes the process take twice as  
long.  But if you hit disk, the performance penalty is around 20%,  
and it drops to around 6% if you feed the items all at once.  Those  
numbers will drop further on larger sorts, which require more merge  
stages and disk i/o.

========================================================
Sort::External version 0.10_5
========================================================
cache_level_one_by_one:           20.503  wallclock secs
mem_thresh_one_by_one:            40.8194 wallclock secs

cache_level_all_at_once:          11.5616 wallclock secs
mem_thresh_all_at_once:           13.9285 wallclock secs

cache_level_hit_disk_one_by_one:  57.9412 wallclock secs
mem_thresh_hit_disk_one_by_one:   69.1299 wallclock secs

cache_level_hit_disk_all_at_once: 45.5628 wallclock secs
mem_thresh_hit_disk_all_at_once:  48.1753 wallclock secs
========================================================

If that's the price to pay for enabling -mem_threshold, I'll buy.

Putting -mem_threshold in Sort::External has actually sped up  
Search::Kinosearch::Kindexer, which had been flushing to disk too  
often under the old scheme of flush-at-10_000-items.  Using - 
mem_threshold took the guesswork out of setting a flush-trigger.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/