APM: Apache mod_perl and large memory data cache

Wayne Walker wwalker at broadq.com
Sun Sep 14 11:47:28 CDT 2003


On Sun, Sep 14, 2003 at 03:29:48AM -0500, Evan Harris wrote:
> 
> I have a mod_perl application that I need to store a large amount of data in
> ram, in the form of a huge multilevel hash.  The problem is that apache
> seems to store this data in each child process, meaning I have apache child
> processes with a larger than 50meg memory footprint, which is way too big.
> 
> It's my understanding that I can use a startup file to preload things,
> but the functions to load the data use many of the request r-> functions,
> which I don't have access to in the startup file.
> 
> I may be able to rewrite my functions not to use the request functions, but
> I don't want to embark on that task unless I know for sure that the
> resulting data structure will reside in shared memory space.  Does anyone
> know for sure?  Is there another (better) way to do this?

If you are on apache 1.x, every child process will have a private copy
of the hash data because Apache 1.x uses fork.

You could get gutsy and try mod_perl under Apache2 and select
multithreading instead of forking, but we just had to downgrade a server
from Apache 2 to Apache 1.3.x because mod_perl was segfaulting children
at random.

I would think about moving the big hash into a helper daemon.  Then you
could get at the data via some network protocol, via a local named pipe,
of possibly via shared memory (I've never tried to do shared mem stuff
in perl).

worst case, you could write a C based Apache Perl module that gives perl
access to a hash in a shared memorty C program that is spawned at Apache
startup.

BUT.  I would recommend one of the following work arounds that might
prevent _any_ code changes.

1. Use internal Proxy stuff in a lightweight non mod_perl apache to
forward the huge request to another apache that is running mod_perl,
does have a huge footprint, but ONLY handles the requests that require
access to the hash data.

or

2. Based on traffic patterns, keep everything in the current setup, but
limit the number of Apache children and make Keep-Alive very short or
non-existant so that fewer children are needed.



-- 

Wayne Walker

www.broadq.com :)  Bringing digital video and audio to the living room



More information about the Austin mailing list