[Pdx-pm] Measuring memory consumption of scalars
marvin at rectangular.com
Tue Jul 26 10:58:34 PDT 2005
David Wheeler writes...
>> Does this help?
Devel::Size has worked out well.
The only downside is that it looks like it fails 1 out of 13 smoke
tests, so that failure will cascade onto Sort::External.
Eric Wilhelm writes...
> Looks like a good source of info and verification, but maybe not
> something that you want to put in your production code:
> "Devel::Size, because of the way it works, can consume a
> amount of memory as it runs. It will use five pointers, two integers,
> and two bytes worth of storage, plus potential alignment and bucket
> overhead, per thing it looks at."
The extra storage overhead turns out not to be all that much.
In the new version of Sort::External, sortex objects track of the
size of the input cache by calling total_size(\@_) every time you feed
I tried two different approaches with a c. 10_000-element source
array (each item a 60-character string): feeding the items to the
sortex object one by one, and feeding them all at once. I set up two
long Benchmark loops, with the one_by_one approach first.
Watching top, I saw the memory consumption of the process stabilize
at 7528k during the one_by_one phase, then rise to a plateau of 8072k
all_at_once phase. That's acceptable.
The performance penalty, as anticipated, diminishes in importance one
tempfiles are required. If you don't hit disk, and you're feeding
items one-by-one, using total_size makes the process take twice as
long. But if you hit disk, the performance penalty is around 20%,
and it drops to around 6% if you feed the items all at once. Those
numbers will drop further on larger sorts, which require more merge
stages and disk i/o.
Sort::External version 0.10_5
cache_level_one_by_one: 20.503 wallclock secs
mem_thresh_one_by_one: 40.8194 wallclock secs
cache_level_all_at_once: 11.5616 wallclock secs
mem_thresh_all_at_once: 13.9285 wallclock secs
cache_level_hit_disk_one_by_one: 57.9412 wallclock secs
mem_thresh_hit_disk_one_by_one: 69.1299 wallclock secs
cache_level_hit_disk_all_at_once: 45.5628 wallclock secs
mem_thresh_hit_disk_all_at_once: 48.1753 wallclock secs
If that's the price to pay for enabling -mem_threshold, I'll buy.
Putting -mem_threshold in Sort::External has actually sped up
Search::Kinosearch::Kindexer, which had been flushing to disk too
often under the old scheme of flush-at-10_000-items. Using -
mem_threshold took the guesswork out of setting a flush-trigger.
More information about the Pdx-pm-list