[VPM] link 'bot' protection
Matt Elrod
matt at elrod.ca
Mon Feb 23 16:13:03 PST 2009
I should think you would want to "throttle" bots by timing
their requests and temporarily banning IPs that exceed a
speed limit. You can specify a preferred delay in your
robots.txt file to give fair warning.
Granted, giving bots a chance to exceed your speed limit
gives them a chance to slurp some of your data, but if
your code blocks them after a dozen or so rapid requests,
they won't get far.
The user-agent variable is easily forged, so speed of
requests is the only reliable way of spotting bots that
I am aware of.
Matt Elrod
Jer A wrote:
> hi all,
>
> I am designing a website service.
>
> how do i prevent automated bots and link scrapers and cross-site scripts
> from access to the site, without hindering the user experience, as well
> as hindering the performance of the host/server/site?
>
> My site is not graphic intensive, and I do not think anyone would be
> interest at grabbing anything that is graphical, only Information/Data.
>
> I have thought of banning ip's by parsing log files, but what should I
> look for that is 'fishy'?
>
> Thanks in advance for all advice/help.
>
> Regards,
> Jeremy
More information about the Victoria-pm
mailing list