[sf-perl] 406 Not Acceptable

Michael Friedman friedman at highwire.stanford.edu
Sat Jul 11 14:09:26 PDT 2009


David,

I wonder if they started blocking the user agent 'libwww-perl/5.816'.  
When my company finds someone illegally spidering our sites, if  
they're using an uncommon user agent string, we often block based on  
that, so that others using the same spidering software can't get in.

Try changing your user agent and see if that helps.

-- Mike
______________________________________________________________________________
Mike Friedman | HighWire Press, Stanford Univ | friedman at highwire.stanford.edu

On Jul 11, 2009, at 2:04 PM, David Alban wrote:

> greetings,
>
> i have a "pingurl" program i've been using for years to monitor
> websites.  all of a sudden (within the last two days), it's been
> alerting me that my site is down (which is not the case).  i suspect
> my hosting provider is mucking with their web server software, but am
> not sure.
>
> basically, i'm doing:
>
>     75 my $ua = LWP::UserAgent->new( env_proxy  => 1,
>     76                               keep_alive => 1,
>     77                               timeout    => 30,
>     78                             );
>     79
>     80 my $response = $ua->get( $url );
>
> in the debugger, while at line 80, i see:
>
>       DB<2> x $ua
>     0  LWP::UserAgent=HASH(0x9c4ab6c)
>        'conn_cache' => LWP::ConnCache=HASH(0x9c4d624)
>           'cc_conns' => ARRAY(0x9c773cc)
>                empty array
>           'cc_limit_total' => 1
>        'def_headers' => HTTP::Headers=HASH(0x9c4d564)
>           'user-agent' => 'libwww-perl/5.816'
>        'handlers' => HASH(0x9c4d6d4)
>           'response_header' => HTTP::Config=ARRAY(0x9c4da44)
>              0  HASH(0x9c4de64)
>                 'callback' => CODE(0x9c4aadc)
>                    ->
> &LWP::UserAgent::__ANON__[/usr/lib/perl5/vendor_perl/5.10.0/LWP/ 
> UserAgent.pm:621]
> in /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm:609-621
>                 'line' =>
> '/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm:622'
>                 'm_media_type' => 'html'
>                 'owner' => 'LWP::UserAgent::parse_head'
>        'max_redirect' => 7
>        'max_size' => undef
>        'no_proxy' => ARRAY(0x98d5784)
>             empty array
>        'protocols_allowed' => undef
>        'protocols_forbidden' => undef
>        'proxy' => HASH(0x9c4d634)
>             empty hash
>        'requests_redirectable' => ARRAY(0x9c46e3c)
>           0  'GET'
>           1  'HEAD'
>        'show_progress' => undef
>        'timeout' => 30
>        'use_eval' => 1
>
> getting to line 80 with LWP::Debug debugging on, i see:
>
>     LWP::UserAgent::new: ()
>     LWP::UserAgent::request: ()
>     LWP::UserAgent::send_request: GET http://extasia.org/
>     LWP::Protocol::http::request: ()
>     LWP::UserAgent::request: Simple response: Not Acceptable
>
> and then:
>
>       DB<5> x $response
>     0  HTTP::Response=HASH(0xa955dc4)
>        '_content' => '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0// 
> EN">
>     <html><head>
>     <title>406 Not Acceptable</title>
>     </head><body>
>     <h1>Not Acceptable</h1>
>     <p>An appropriate representation of the requested resource /
> could not be found on this server.</p>
>     <p>Additionally, a 404 Not Found
>     error was encountered while trying to use an ErrorDocument to
> handle the request.</p>
>     <hr>
>     <address>Apache/2.2.11 (Unix) mod_ssl/2.2.11
> OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4
> FrontPage/5.0.2.2635 mod_jk/1.2.25 Server at extasia.org Port
> 80</address>
>     </body></html>
>     '
>        '_headers' => HTTP::Headers=HASH(0xa955db4)
>           'client-date' => 'Sat, 11 Jul 2009 20:58:08 GMT'
>           'client-peer' => '208.83.209.13:80'
>           'client-response-num' => 1
>           'connection' => 'close'
>           'content-length' => 564
>           'content-type' => 'text/html; charset=iso-8859-1'
>           'date' => 'Sat, 11 Jul 2009 20:58:03 GMT'
>           'server' => 'Apache/2.2.11 (Unix) mod_ssl/2.2.11
> OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4
> FrontPage/5.0.2.2635 mod_jk/1.2.25'
>           'title' => '406 Not Acceptable'
>        '_msg' => 'Not Acceptable'
>        '_protocol' => 'HTTP/1.1'
>        '_rc' => 406
>        '_request' => HTTP::Request=HASH(0xa56ba5c)
>           '_content' => ''
>           '_headers' => HTTP::Headers=HASH(0xa56ba4c)
>              'user-agent' => 'libwww-perl/5.816'
>           '_method' => 'GET'
>           '_uri' => URI::http=SCALAR(0xa36de84)
>              -> 'http://extasia.org/'
>           '_uri_canonical' => URI::http=SCALAR(0xa36de84)
>              -> REUSED_ADDRESS
>        'handlers' => HASH(0xa98ec2c)
>           'response_data' => ARRAY(0xa9742ac)
>              0  CODE(0xa9743bc)
>                 ->
> &LWP::UserAgent::__ANON__[/usr/lib/perl5/vendor_perl/5.10.0/LWP/ 
> UserAgent.pm:619]
> in /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm:616-619
>              1  CODE(0xa2c087c)
>                 ->
> &LWP::Protocol::__ANON__[/usr/lib/perl5/vendor_perl/5.10.0/LWP/ 
> Protocol.pm:131]
> in /usr/lib/perl5/vendor_perl/5.10.0/LWP/Protocol.pm:128-131
>
> i found http://www.checkupdown.com/status/E406.html and other internet
> resources that describe the 406 response, but i'm not sure how to
> resolve this problem.  i could easily substitute a system() invocation
> of wget for use of LWP (firefox and wget both successfully fetch the
> index page of my site), but i want to understand what has broken this
> after years of successful use.  do i need to add some type of header
> indicating "i'll take whatever wacky content you wanna give me"?
>
> thanks for any assistance,
> david
> -- 
> Live in a world of your own, but always welcome visitors.
> _______________________________________________
> SanFrancisco-pm mailing list
> SanFrancisco-pm at pm.org
> http://mail.pm.org/mailman/listinfo/sanfrancisco-pm



More information about the SanFrancisco-pm mailing list