[Phoenix-pm] Multi-level hash - quick key reference?

Scott Walters scott at illogics.org
Wed Jun 7 16:14:36 PDT 2006 

I should point out that hash keys as stored in the hash use a shared 
string table.  If you're really worried about memory though, use an
mldbm module and go multi-threaded to avoid performance loss due to
cache misses for disc blocks.  It isn't much harder to use the disc
than RAM, depending on what you're storing.

-scott

On  0, "Metz, Bobby W, WWCS" <bwmetz at att.com> wrote:
> For anyone interested...I'd never looked this up before and appears my
> memory usage concern is unfounded:
> 
> "The key structure, though, is of variable length because the key string
> is of variable length, so perl has to ask the system for a memory
> allocation for each key. The base size of this structure is 8 or 16
> bytes (once again, depending on whether you're on a 32 bit or 64 bit
> system) plus the string length plus two bytes.
> 
> Since this memory has to be allocated from the system there's the malloc
> size-field overhead (4 or 8 bytes) plus the alignment bytes (0 to 7,
> depending on your system and the key length) that get added on to the
> chunk perl requests. If the key is only 1 character, and you're on a 32
> bit system, the allocation will be 16 bytes. If the key is 7 characters
> then the allocation is 24 bytes on a 32 bit system. If you're on a 64
> bit system the numbers get even larger."
> 
> Taken from http://search.cpan.org/~dsugal/Devel-Size-0.64/Size.pm
> 
> so, if I'm reading this correctly:
> 
> $hash{".1.2.3.4"} --> approx 22 to 29 bytes (sorry not sure how many
> alignment bytes would be needed)
> 
> vs.
> 
> $hash{1}{2}{3}{4} --> approx 16 * 4 or 64 bytes
> 
> So, I guess flattening doesn't have the memory trade-off I was worried
> about since long hash keys have less impact on memory allocation than
> splitting it into nested single char keys.  If I'm reading this wrong,
> someone please correct me.
> 
> -----Original Message-----
> From: phoenix-pm-bounces+bwmetz=att.com at pm.org
> [mailto:phoenix-pm-bounces+bwmetz=att.com at pm.org]On Behalf Of Metz,
> Bobby W, WWCS
> Sent: Wednesday, June 07, 2006 1:03 PM
> To: Brock
> Cc: phoenix-pm at pm.org
> Subject: Re: [Phoenix-pm] Multi-level hash - quick key reference?
> 
> 
> Brock,
> 	May have to give yours a go after all...just dawned on me that
> flattening the index will significantly increase storage due to the
> duplication of index characters.  Some could have several thousand
> values so if it were say six keys deep times a thousand values, it would
> add up over time.
> 
> Bobby
> 
> -----Original Message-----
> From: phoenix-pm-bounces+bwmetz=att.com at pm.org
> [mailto:phoenix-pm-bounces+bwmetz=att.com at pm.org]On Behalf Of Metz,
> Bobby W, WWCS
> Sent: Wednesday, June 07, 2006 1:00 PM
> To: Brock
> Cc: phoenix-pm at pm.org
> Subject: Re: [Phoenix-pm] Multi-level hash - quick key reference?
> 
> 
> Thanks Brock.  
> 	After sending that message I thought about flattening it as
> well...a big resounding "Duh" escaped my lips I must admit.  I
> considered eval but quickly squashed that idea.  I should have thought
> of a shift method...so thanks, I like that.  I think I need to study the
> data source some more as flattening will likely be the simplest method
> since each index will have at most two values.  I oversimplified a bit
> on the example.
> 
> $hash{.1.2.3.4}{name} & $hash{.1.2.3.4}{value}
> 
> And since the names will never change, I'll likely just have one hash
> for those and an array of hashes for the values.
> 
> Thanks for the quick response...very much appreciated.
> 
> Bobby
> 
> -----Original Message-----
> From: Brock [mailto:awwaiid at thelackthereof.org]
> Sent: Wednesday, June 07, 2006 12:52 PM
> To: Metz, Bobby W, WWCS
> Cc: phoenix-pm at pm.org
> Subject: Re: [Phoenix-pm] Multi-level hash - quick key reference?
> 
> 
> Heres a couple ideas. One, you could just flatten your hash -- use
> 
>   $hash->{1.2.3} = "Hello World";
>   $hash->{1.2.3.4.5} = "the quick brown fox";
> 
> but, failing that,
> 
> use strict;
> my $hash = {};
> my $str = ".1.2.3\tHello World";
> my ($nums,$text) = split /\t/,$str;
> my @indexes = split /\./, $nums;
> shift @indexes; # Get rid of the first ""
> my $cur = $hash;
> while(@indexes) {
>   my $index = shift @indexes;
>   if(!@indexes) { # this is the last one
>     $cur->{$index} = $text;
>   } else {
>     # We might need to vivify
>     $cur->{$index} = {} unless defined $cur->{$index};
>     $cur = $cur->{$index};
>   }
> }
> use Data::Dumper;
> print "Result: " . Dumper($hash) . "\n";
> 
> might help. Simplification, anyone? I suppose you could use evil, er,
> eval.
> 
> --Brock
> 
> On 2006.06.07.14.31, Metz, Bobby W, WWCS wrote:
> | All,
> | 	I'm looking for a quicker method of accessing elements in a
> multi-level hash.  For example, the hash will have values like so
> | 
> | $hash{1}{2}{3} = "Hello World";
> | $hash{1}{3}{4}{5} = "the quick brown fox";
> | 
> | The input to the program comes from another I can't control which will
> ouput values like so:
> | 
> | .1.2.3<tab>Hello World
> | .1.3.4.5<tab>the quick brown fox
> | 
> | Parsing the values is cake, but what I'm looking for is a faster way
> to get values in/out of the hash using the ".1.2.3".  Is there some
> method that allows for arbitrary hash levels without coding using for
> loops?  I mean clearly recursion would solve this but I was hoping there
> was something more simple.  Or is recursion the only answer.  I was
> hoping for something like this, but I know this example doesn't work.
> I've searched for a bit but I must be asking the right questions of
> mighty google.
> | 
> | $index = ".1.2.3";
> | $index =~ s/^\.//g;
> | $index =~ s/\./}{/g;
> | print $hash{$index} . "\n";
> | 
> | Thoughts?
> | 
> | Thanks,
> | 
> | Bobby
> | _______________________________________________
> | Phoenix-pm mailing list
> | Phoenix-pm at pm.org
> | http://mail.pm.org/mailman/listinfo/phoenix-pm
> _______________________________________________
> Phoenix-pm mailing list
> Phoenix-pm at pm.org
> http://mail.pm.org/mailman/listinfo/phoenix-pm
> _______________________________________________
> Phoenix-pm mailing list
> Phoenix-pm at pm.org
> http://mail.pm.org/mailman/listinfo/phoenix-pm
> _______________________________________________
> Phoenix-pm mailing list
> Phoenix-pm at pm.org
> http://mail.pm.org/mailman/listinfo/phoenix-pm

More information about the Phoenix-pm mailing list