[VPM] link 'bot' protection

Jer A jeremygwa at hotmail.com
Mon Feb 23 16:57:39 PST 2009



If I were to ban by ip, what if it were only one bad machine in a large network behind a router.....will it block the entire network?

Thanks again,

Jeremy

> Date: Mon, 23 Feb 2009 16:13:03 -0800
> From: matt at elrod.ca
> To: jeremygwa at hotmail.com
> CC: victoria-pm at pm.org
> Subject: Re: [VPM] link 'bot' protection
> 
> 
> I should think you would want to "throttle" bots by timing
> their requests and temporarily banning IPs that exceed a
> speed limit.  You can specify a preferred delay in your
> robots.txt file to give fair warning.
> 
> Granted, giving bots a chance to exceed your speed limit
> gives them a chance to slurp some of your data, but if
> your code blocks them after a dozen or so rapid requests,
> they won't get far.
> 
> The user-agent variable is easily forged, so speed of
> requests is the only reliable way of spotting bots that
> I am aware of.
> 
> Matt Elrod
> 
> Jer A wrote:
> > hi all,
> > 
> > I am designing a website service.
> > 
> > how do i prevent automated bots and link scrapers and cross-site scripts 
> > from access to the site, without hindering the user experience, as well 
> > as hindering the performance of the host/server/site?
> > 
> > My site is not graphic intensive, and I do not think anyone would be 
> > interest at grabbing anything that is graphical, only Information/Data.
> > 
> > I have thought of banning ip's by parsing log files, but what should I 
> > look for that is 'fishy'?
> > 
> > Thanks in advance for all advice/help.
> > 
> > Regards,
> > Jeremy
> 

_________________________________________________________________
The new Windows Live Messenger. You don’t want to miss this.
http://www.microsoft.com/windows/windowslive/products/messenger.aspx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.pm.org/pipermail/victoria-pm/attachments/20090223/8cd8862f/attachment.html>


More information about the Victoria-pm mailing list