[VPM] link 'bot' protection

[Matt Elrod]

I should think you would want to "throttle" bots by timing
their requests and temporarily banning IPs that exceed a
speed limit.  You can specify a preferred delay in your
robots.txt file to give fair warning.

Granted, giving bots a chance to exceed your speed limit
gives them a chance to slurp some of your data, but if
your code blocks them after a dozen or so rapid requests,
they won't get far.

The user-agent variable is easily forged, so speed of
requests is the only reliable way of spotting bots that
I am aware of.

Matt Elrod

Jer A wrote:
> hi all,
> 
> I am designing a website service.
> 
> how do i prevent automated bots and link scrapers and cross-site scripts 
> from access to the site, without hindering the user experience, as well 
> as hindering the performance of the host/server/site?
> 
> My site is not graphic intensive, and I do not think anyone would be 
> interest at grabbing anything that is graphical, only Information/Data.
> 
> I have thought of banning ip's by parsing log files, but what should I 
> look for that is 'fishy'?
> 
> Thanks in advance for all advice/help.
> 
> Regards,
> Jeremy