[Melbourne-pm] Mod_perl2

[Scott Penrose]

On 14/09/2006, at 13:12, Andrew Speer wrote:

> On Thu, September 14, 2006 11:35 am, Scott Penrose wrote:
>> Hey Guys
>>
>> .. snip ..
>>
>> The sort of topics I would like to cover are:
>>
>> * Why should you use mod_perl
>> * Why should you use mod_perl2
>> * Why should you not use mod_perl
>> * Processing Parameters, Environment Variables and Headers - in
>> Content and Filter handlers (and other stages)
>>
>
> Hi,
>
> I am from Adelaide, so was not able to make the meeting. It may  
> have been
> covered, but one of the issues I seen when using mod_perl(2) is memory
> usage per connection.
>
> When using Apache with mod_perl processes can easily grow to 10M+ (or
> more, depending on the app). If the site/app suddenly becomes  
> subject to a
> high load Apache will spawn more processes to service the load.  
> Unless you
> have given very careful consideration to memory utilisation the  
> server can
> easily be swamped by Apache processes needing more memory than is
> physically available. The server starts swapping and performance  
> suffers
> badly.

I think this is to do with Perl, not mod_perl. Our stand alone perl  
daemons for web applications, once loaded all the necessary modules  
often start at around 80MB.

However, each fork or thread consumes virtually no more memory -  
unless you are using an older Digital Unix, or Solaris - as they are  
always spawned copy on write - which means most content does not get  
effected.

I have found a 512MB machine with a 100MB Apache process can easily  
be forked 50+ times without hitting swap.

However, what you are talking about is correct - memory is an issue,  
but my thought on the space is that a stand alone application causes  
the same memory consumption.

> There are ways and means around this problem, but they all seem a bit
> kludgy - front-end proxy servers, multiple Apache servers (Apache with
> mod_perl for dynamic content, straight Apache for static content).

I have seen this often, especially using two apache servers, one for  
static content the other for dynamic.
I have never needed to do this, and question some of the assumptions  
- e.g. the time to fork an apache daemon is fairly constant, and not  
usually affected by process size, and if you hit only the internal C  
default handler for static content, should be about as fast.
I suspect that if you add in a bad handler or config handler, or  
other startup type modules, then you are running perl code on fork  
that may not be required and that people do this way for convenience.

What we need is some work proving one way or another what is  
happening - not too hard to do, just requires a good day from people.

> If your main need is content-phase dynamic content (a big part of most
> mod_perl apps) you may want to have a look at lighttpd + fastCGI  
> and the
> CPAN FastCGI module. lighttpd spawns as many Perl FastCGI processes  
> and
> you want, and only sends requests to them if dynamic content  
> generations
> is required - as opposed to Apache, which in the "default" mod_perl  
> config
> will handle both static and dynamic content - ie you may have a 10M+
> process tied up sending down a JPEG or CSS file.

My experience with the above is you get the same memory footprint. As  
I said, it is not mod_perl, it is Perl that is large.

I am amused by the 10M process too because our process are normally  
about 300MB using perl, and Java Tomcat starts at about 800MB+ - we  
have servers that one process is GB for TomCat - but it still runs at  
a few hundred hits a second. Although it is slow not because of the  
process size but because of what it is doing.

> If you want to do tricky stuff outside the content generation phase  
> then
> mod_perl is still one of the best/easiest ways to extend Apache  
> with Perl.
>
> Anyway, there are heaps of resources on the Internet discussing  
> these sort
> of issues, but a dedicated mod_perl wiki does sound like a good idea.

Agreed. It would be nice to have these sort of questions, maybe some  
anecdotes (like we have both written here) and then some tests to  
prove the items. All easy if we had a place to document.

> Now for a shameless plug of my mod_perl framework. Have a look at
> http://webdyne.org/ for my implementation of YAPWF (Yet Another  
> Perl Web
> Framework). It started off as a mod_perl implementation, but now runs
> under IIS/lighttpd and FastCGI also.

Scott