[VPM] - data structures, performance and memory

[Darren Duncan]

At 8:34 PM -0800 11/9/06, Jer A wrote:
>hi all perl gurus,
>I am working on two projects where I can use some of your advice.

I'll see what I can do here.

>project 1: I am volunteering for a non-profit organization putting a printed
>"information directory" on their website. It will consist of a search
>engine, and an Index of categories, where one can narrow down, and browse,
>if they do not wish to do a search. Due, to funding and other restrictions,
>they cannot host an sql database, xml or webservices, as they use SHAW
>hosting, So I will be doing this project using a flat file approach. <snip>

The simplest solution I see, if your content is static and 
infrequently changes, and you are staying with Shaw, is to just 
pre-generate the index and content pages as static HTML and just dump 
it on the server.  Any code will run on your own machine, where you 
can use a database.  You can then use a specially crafted Google 
link, one containing "site:" in the query, to just let Google provide 
the search mechanism for your site.  Lots of small (and larger) sites 
use Google to do their search engine for them.

If your content is changed regularly and/or is interactive, I suggest 
getting a different host.  Shaw costs at least $20-50 per month, 
AFAIK, and a dedicated web host with databases and such can be had 
for as low as $10 per month, and you get your own domain name with 
those too.

If neither of these is suitable for some reason, then please give 
more information about how much data you have, what kind of data it 
is, and about how much traffic you get, plus what features Shaw does 
give you.

>project 2: I am working on a project where I need to store data in memory.
>What are the memory requirements of certain structures eg. hash,array,array
>of anon hashes,array of packed data,array of strings, array of objects etc.
>
>suppose I have a hundred records (structures) holding eg. age,name,address
>etc. what is the best way of storing this data live in memory for a long
>period time, for performance and efficiency.
>An array of anon hashes is pretty, but may not be the best for performance,
>and mem usage...am i right? Considering this, how can I store records with
>elements that can be of different record types, saving the most amount of
>memory as possible......can I do this with pack and unpack, I am not
>familiar with this, but do you think s more efficient, than an array of
>element delimited strings? or what if each record is an instantiated object
>with properties?
>
>Thanks in advance for your advice and help.
>
>   -Jeremy A.

I think you may be getting into a "premature optimization" matter. 
If you have only a hundred records of eg age,name,address, those will 
all take up maybe 1 kilobyte at most of memory, which is negligible. 
Don't bother fussing about memory unless you're storing tens of 
thousands or more of such records, or you're working on an embedded 
system.  Just do whatever is easiest to program, as programmer 
efficiency is usually more important than any other kind.  Use hashes 
if you're looking data up directly, such as by the person's name, and 
arrays only when you plan to go through them sequentially.  Or you 
can include multiple references to the same data structures if you 
need to.

-- Darren Duncan