[Columbus-pm] Fwd: MLDBM-like module

[Michael Frager]

Great question!

Tie::DBI can only store one level of data. If a hash key contains a nested
data structure, then that structure is just serialized with Storable and
stored in one row.

The module I've written is different in a few ways. First, it supports both
hashes and arrays (it's really two modules, one for each). Second, it's
totally recursive: if the root hash contains nested structures, those
structures are automatically stored in the same way as the root. This means
that key/value pairs and array elements of structures deep within the tree
will also each have their own rows.

Here is the advantage of doing this: SPEED!

When I go to read just a few keys deep within a structure, the algorithm
only has to follow the path through the tree to those specific keys (or
array indexes). It can leave the rest of the data in the database. It
doesn't have to pull the whole thing and de-serialize all of it.

Or, if I want an entire hash, then a different SQL query is used, returning
the entire hash's key/value pairs in one SQL query. Also, the entire table
schema is properly indexed in the SQL database, so lookups of individual
values and entire structures are all done using index lookups and not table
scans.

Finally, caching is employed, so data only has to be read from the SQL
server once.


-Mike

On Mon, Nov 16, 2009 at 8:58 PM, Jonathan Hogue <jon at hogue.org> wrote:

> I would love to learn more! How is it different from Tie::DBI? :-)
>
>
> 2009/11/16 Michael Frager <mike at dialyourleads.com>:
> > Hello,
> >
> > The big difference between my module and DBM::Deep is that my module uses
> a
> > SQL database (like MySQL) instead of flat files. DBM::Deep has
> implemented
> > its own locking and concurrency system that is not likely to be a robust
> as
> > a real SQL database.
> >
> > Other than that the concepts are very similar. Basically, you do what you
> > want to your data structures and the results are transparently and
> > persistently stored for you.
> >
> >
> > -Mike
> >
> >
> > On Mon, Nov 16, 2009 at 5:27 PM, Rob Kinyon <rob.kinyon at gmail.com>
> wrote:
> >>
> >> I'd be very interested in seeing how this and DBM::Deep could play
> >> together.
> >>
> >> Rob
> >>
> >> 2009/11/16 Michael Frager <mike at dialyourleads.com>:
> >> > Hello (again) Columbus.pm,
> >> >
> >> > I have developed a module that is somewhat like MLDBM. In case you're
> >> > not
> >> > familiar with MLDBM, this module will take a hash and synchronize its
> >> > contents with a DB file for persistent storage. Unlike a regular hash
> >> > tied
> >> > to a DB file, MLDBM will save entire data structures that are
> contained
> >> > within the hash keys, serializing them on the fly.
> >> >
> >> > I've taken that concept a little bit further and developed a module
> that
> >> > will synchronize both hashes and arrays with a SQL database. Each
> >> > element of
> >> > the data structure being stored can either be stored as its own row or
> >> > its
> >> > entire contents serialized (similar to MLDBM). The process is
> recursive
> >> > so
> >> > entire structures can be stored this way.
> >> >
> >> > The upshot is that you can take an entire Perl data structure, made of
> >> > any
> >> > combination of hashes and arrays, and just place it into a SQL
> database.
> >> > It
> >> > works no matter how deep the structure is and without the need for a
> >> > custom
> >> > SQL schema. Since Perl's "tie" function is use, once the structure is
> >> > stored, it can be manipulated like any variable, except that changes
> >> > will be
> >> > transparently stored in the SQL database.
> >> >
> >> > I'm in the processes of preparing this module for CPAN, I was
> wondering
> >> > if
> >> > there is anyone in the Columbus.pm group who might be interested in
> >> > learning
> >> > about it?
> >> >
> >> >
> >> >
> >> > Thanks,
> >> >
> >> >
> >> > Michael K Frager, Software Developer
> >> >
> >> > ~ Dial Your Leads ~
> >> >
> >> > [+] Phone:    (888) 310-4474
> >> > [+] Website: https://www.dialyourleads.com/
> >> >
> >> >
> >> > _______________________________________________
> >> > Columbus-pm mailing list
> >> > http://columbus.pm.org/
> >> > Columbus-pm at pm.org
> >> > http://mail.pm.org/mailman/listinfo/columbus-pm
> >> >
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Rob Kinyon
> >
> >
> >
> > --
> >
> > Michael K Frager, Software Developer
> >
> > ~ Dial Your Leads ~
> >
> > [+] Phone:    (888) 310-4474
> > [+] Website: https://www.dialyourleads.com/
> >
> >
> > _______________________________________________
> > Columbus-pm mailing list
> > http://columbus.pm.org/
> > Columbus-pm at pm.org
> > http://mail.pm.org/mailman/listinfo/columbus-pm
> >
>



-- 

Michael K Frager, Software Developer

~ Dial Your Leads ~

[+] Phone:    (888) 310-4474
[+] Website: https://www.dialyourleads.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.pm.org/pipermail/columbus-pm/attachments/20091116/349927ac/attachment-0001.html>