[sf-perl] 406 Not Acceptable

[Michael Friedman]

Hey, look at this undocumented feature!

 From LWP::UserAgent, every version I could find:

sub _agent       { "libwww-perl/$LWP::VERSION" }

sub agent {
     my $self = shift;
     my $old = $self->{agent};
     if (@_) {
         my $agent = shift;
         $agent .= $self->_agent if $agent && $agent =~ /\s+$/;
         $self->{agent} = $agent;
     }
     $old;
}


Did you perhaps leave a space on the end of your new agent string?
Wow, this looks remarkably annoying. I wonder why the author did that?
It's obviously intentional, as it's been continuously in service from  
at least version 2.36 to 5.829 (the latest).

-- Mike
______________________________________________________________________________
Mike Friedman | HighWire Press, Stanford Univ | friedman at highwire.stanford.edu

On Jul 11, 2009, at 2:26 PM, David Alban wrote:

> thanks, mike.
>
> it looks as though any agent string i specify is prepended to the
> libwww-perl/...  string, and doesn't replace it.
>
>   'def_headers' => HTTP::Headers=HASH(0xa32bf64)
>      'user-agent' => 'Firefox/3.0.11 libwww-perl/5.816'
>
> and it still comes back with 406:
>
>      'server' => 'Apache/2.2.11 (Unix) mod_ssl/2.2.11
> OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4
> FrontPage/5.0.2.2635 mod_jk/1.2.25'
>      'title' => '406 Not Acceptable'
>   '_msg' => 'Not Acceptable'
>   '_protocol' => 'HTTP/1.1'
>   '_rc' => 406
>
>
> On Sat, Jul 11, 2009 at 2:09 PM, Michael
> Friedman<friedman at highwire.stanford.edu> wrote:
>> I wonder if they started blocking the user agent 'libwww-perl/ 
>> 5.816'. When
>> my company finds someone illegally spidering our sites, if they're  
>> using an
>> uncommon user agent string, we often block based on that, so that  
>> others
>> using the same spidering software can't get in.
>>
>> Try changing your user agent and see if that helps.
>
> -- 
> Live in a world of your own, but always welcome visitors.
> _______________________________________________
> SanFrancisco-pm mailing list
> SanFrancisco-pm at pm.org
> http://mail.pm.org/mailman/listinfo/sanfrancisco-pm