[VPM] link 'bot' protection
Jer A
jeremygwa at hotmail.com
Mon Feb 23 16:57:39 PST 2009
If I were to ban by ip, what if it were only one bad machine in a large network behind a router.....will it block the entire network?
Thanks again,
Jeremy
> Date: Mon, 23 Feb 2009 16:13:03 -0800
> From: matt at elrod.ca
> To: jeremygwa at hotmail.com
> CC: victoria-pm at pm.org
> Subject: Re: [VPM] link 'bot' protection
>
>
> I should think you would want to "throttle" bots by timing
> their requests and temporarily banning IPs that exceed a
> speed limit. You can specify a preferred delay in your
> robots.txt file to give fair warning.
>
> Granted, giving bots a chance to exceed your speed limit
> gives them a chance to slurp some of your data, but if
> your code blocks them after a dozen or so rapid requests,
> they won't get far.
>
> The user-agent variable is easily forged, so speed of
> requests is the only reliable way of spotting bots that
> I am aware of.
>
> Matt Elrod
>
> Jer A wrote:
> > hi all,
> >
> > I am designing a website service.
> >
> > how do i prevent automated bots and link scrapers and cross-site scripts
> > from access to the site, without hindering the user experience, as well
> > as hindering the performance of the host/server/site?
> >
> > My site is not graphic intensive, and I do not think anyone would be
> > interest at grabbing anything that is graphical, only Information/Data.
> >
> > I have thought of banning ip's by parsing log files, but what should I
> > look for that is 'fishy'?
> >
> > Thanks in advance for all advice/help.
> >
> > Regards,
> > Jeremy
>
_________________________________________________________________
The new Windows Live Messenger. You don’t want to miss this.
http://www.microsoft.com/windows/windowslive/products/messenger.aspx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.pm.org/pipermail/victoria-pm/attachments/20090223/8cd8862f/attachment.html>
More information about the Victoria-pm
mailing list