Phoenix.pm: keep-alives with RobotUA

[Michael Friedman]

David,

On Dec 5, 2003, at 7:04 AM, David A. Sinck wrote:
> I'm guessing you're not using a pentium 90 too.  :-)

Heh. No, not Pentium 90s. More like 20 Sun E450s plus a multi-terabyte 
disk server. We've looked into Linux, though, and it could actually 
handle our traffic pretty well, if we had enough standard Dell (or 
similar) machines to host everything.

> \_ Now, we've taken steps to avoid such behavior. If you hit one of our
> \_ sites that fast, we'll block you from getting any pages within a
> \_ second.
>
> This sounds like a juicy bit, if you care to illucidate?

We wrote an Apache module (in C, but the prototype was in mod_perl) 
that keeps a log of traffic by user. If the module sees too many 
requests too fast from a single machine, single browser, or single 
proxy server, it aborts the Apache delivery process and sends back a 
warning page instead. Then it keeps a list of people who have been 
blocked and slowly lets the blocks expire over time.

Since the module is high up in Apache's chain, it takes less effort to 
block someone than to serve their requested page, thus reducing the 
load on our machine and (since the warning page is small with no 
graphics) reducing the network traffic.

You can still flood the server if you try hard enough with a DDOS, but 
it takes more effort than we have enemies, if you know what I mean.

-- Mike
---------------------------------------------------------------------
Michael Friedman                  HighWire Press, Stanford Southwest
Phone: 480-456-0880                                   Tempe, Arizona
FAX:   270-721-8034                  <friedman at highwire.stanford.edu>
---------------------------------------------------------------------